Nov 02 01:23:57.900726 ip-10-0-129-6 systemd[1]: Starting Kubernetes Kubelet... Nov 02 01:23:58.284133 ip-10-0-129-6 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 01:23:58.284133 ip-10-0-129-6 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 01:23:58.284133 ip-10-0-129-6 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 01:23:58.284133 ip-10-0-129-6 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 01:23:58.289805 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289726 2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:23:58.289805 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289805 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:23:58.289873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289809 2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:23:58.289873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289812 2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:23:58.289873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289815 2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:23:58.289873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289818 2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:23:58.289873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289821 2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:23:58.289873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289823 2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:23:58.289873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289827 2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:23:58.289873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289829 2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:23:58.289873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289832 2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:23:58.289873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289834 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:23:58.289873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289842 2567 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:23:58.289873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289845 2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:23:58.289873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289848 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:23:58.289873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289850 2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:23:58.289873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289853 2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:23:58.289873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289855 2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:23:58.289873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289858 2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:23:58.289873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289861 2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:23:58.289873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289863 2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:23:58.289873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289866 2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:23:58.290341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289868 2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:23:58.290341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289871 2567 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:23:58.290341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289874 2567 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:23:58.290341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289876 2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:23:58.290341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289879 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:23:58.290341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289882 2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:23:58.290341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289885 2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:23:58.290341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289889 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:23:58.290341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289892 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:23:58.290341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289894 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:23:58.290341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289897 2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:23:58.290341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289899 2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:23:58.290341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289902 2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:23:58.290341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289905 2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:23:58.290341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289907 2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:23:58.290341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289909 2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:23:58.290341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289912 2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:23:58.290341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289915 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:23:58.290341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289917 2567 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:23:58.290341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289920 2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:23:58.290829 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289923 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:23:58.290829 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289925 2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:23:58.290829 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289930 2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:23:58.290829 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289934 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:23:58.290829 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289938 2567 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:23:58.290829 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289940 2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:23:58.290829 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289943 2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:23:58.290829 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289946 2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:23:58.290829 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289948 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:23:58.290829 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289951 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:23:58.290829 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289954 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:23:58.290829 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289957 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:23:58.290829 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289960 2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:23:58.290829 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289963 2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:23:58.290829 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289966 2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:23:58.290829 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289968 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:23:58.290829 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289971 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:23:58.290829 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289974 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:23:58.290829 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289977 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:23:58.291281 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289980 2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:23:58.291281 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289983 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:23:58.291281 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289985 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:23:58.291281 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289988 2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:23:58.291281 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289991 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:23:58.291281 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289993 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:23:58.291281 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289996 2567 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:23:58.291281 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.289999 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:23:58.291281 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290001 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:23:58.291281 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290004 2567 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:23:58.291281 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290007 2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:23:58.291281 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290009 2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:23:58.291281 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290013 2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:23:58.291281 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290015 2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:23:58.291281 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290018 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:23:58.291281 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290021 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:23:58.291281 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290023 2567 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:23:58.291281 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290026 2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:23:58.291281 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290029 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:23:58.291281 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290032 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:23:58.291773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290034 2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:23:58.291773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290037 2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:23:58.291773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290040 2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:23:58.291773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290043 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:23:58.291773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290045 2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:23:58.291773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290047 2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:23:58.291773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290051 2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:23:58.291773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290053 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:23:58.291773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290056 2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:23:58.291773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290058 2567 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:23:58.291773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290061 2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:23:58.291773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290064 2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:23:58.291773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290066 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:23:58.291773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290069 2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:23:58.291773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290073 2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:23:58.291773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290077 2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:23:58.291773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290081 2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:23:58.291773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290084 2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:23:58.291773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290089 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:23:58.292310 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290094 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:23:58.292310 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290100 2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:23:58.292310 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290103 2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:23:58.292310 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290106 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:23:58.292310 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290108 2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:23:58.292310 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290112 2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:23:58.292310 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290877 2567 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:23:58.292310 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290885 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:23:58.292310 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290889 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:23:58.292310 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290892 2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:23:58.292310 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290895 2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:23:58.292310 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290898 2567 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:23:58.292310 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290900 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:23:58.292310 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290903 2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:23:58.292310 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290906 2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:23:58.292310 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290908 2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:23:58.292310 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290911 2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:23:58.292310 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290915 2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:23:58.292310 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290917 2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:23:58.292310 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290921 2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:23:58.292804 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290924 2567 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:23:58.292804 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290926 2567 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:23:58.292804 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290929 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:23:58.292804 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290931 2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:23:58.292804 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290934 2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:23:58.292804 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290937 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:23:58.292804 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290939 2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:23:58.292804 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290942 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:23:58.292804 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290945 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:23:58.292804 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290948 2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:23:58.292804 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290950 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:23:58.292804 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290953 2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:23:58.292804 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290955 2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:23:58.292804 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290958 2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:23:58.292804 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290961 2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:23:58.292804 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290964 2567 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:23:58.292804 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290966 2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:23:58.292804 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290969 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:23:58.292804 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290971 2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:23:58.293478 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290976 2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:23:58.293478 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290980 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:23:58.293478 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290983 2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:23:58.293478 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290986 2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:23:58.293478 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290989 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:23:58.293478 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290991 2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:23:58.293478 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290994 2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:23:58.293478 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290996 2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:23:58.293478 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.290999 2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:23:58.293478 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291001 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:23:58.293478 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291004 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:23:58.293478 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291007 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:23:58.293478 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291010 2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:23:58.293478 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291012 2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:23:58.293478 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291015 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:23:58.293478 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291018 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:23:58.293478 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291021 2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:23:58.293478 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291023 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:23:58.293478 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291026 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:23:58.294171 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291028 2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:23:58.294171 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291031 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:23:58.294171 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291034 2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:23:58.294171 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291037 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:23:58.294171 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291039 2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:23:58.294171 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291042 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:23:58.294171 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291044 2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:23:58.294171 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291047 2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:23:58.294171 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291049 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:23:58.294171 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291052 2567 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:23:58.294171 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291054 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:23:58.294171 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291057 2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:23:58.294171 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291060 2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:23:58.294171 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291062 2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:23:58.294171 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291065 2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:23:58.294171 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291068 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:23:58.294171 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291070 2567 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:23:58.294171 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291073 2567 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:23:58.294171 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291075 2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:23:58.294171 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291078 2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:23:58.294669 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291081 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:23:58.294669 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291083 2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:23:58.294669 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291086 2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:23:58.294669 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291088 2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:23:58.294669 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291091 2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:23:58.294669 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291095 2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:23:58.294669 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291099 2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:23:58.294669 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291102 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:23:58.294669 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291105 2567 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:23:58.294669 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291107 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:23:58.294669 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291110 2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:23:58.294669 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291112 2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:23:58.294669 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291115 2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:23:58.294669 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291117 2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:23:58.294669 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291119 2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:23:58.294669 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291122 2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:23:58.294669 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291125 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:23:58.294669 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291128 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:23:58.294669 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291130 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:23:58.295143 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291133 2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:23:58.295143 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291135 2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:23:58.295143 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291138 2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:23:58.295143 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291140 2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:23:58.295143 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291143 2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:23:58.295143 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291145 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:23:58.295143 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291148 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:23:58.295143 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291150 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:23:58.295143 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291153 2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:23:58.295143 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291156 2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:23:58.295143 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291159 2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:23:58.295143 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291161 2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:23:58.295143 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291164 2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:23:58.295143 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291166 2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:23:58.295143 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291170 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:23:58.295143 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291658 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:23:58.295143 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291663 2567 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:23:58.295143 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291666 2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:23:58.295143 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291669 2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:23:58.295143 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291672 2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:23:58.295629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291675 2567 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:23:58.295629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291677 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:23:58.295629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291680 2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:23:58.295629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291683 2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:23:58.295629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291686 2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:23:58.295629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291689 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:23:58.295629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291691 2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:23:58.295629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291694 2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:23:58.295629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291697 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:23:58.295629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291699 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:23:58.295629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291702 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:23:58.295629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291704 2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:23:58.295629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291707 2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:23:58.295629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291710 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:23:58.295629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291712 2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:23:58.295629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291715 2567 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:23:58.295629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291718 2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:23:58.295629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291720 2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:23:58.295629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291723 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:23:58.296081 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291726 2567 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:23:58.296081 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291728 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:23:58.296081 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291731 2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:23:58.296081 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291734 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:23:58.296081 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291736 2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:23:58.296081 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291739 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:23:58.296081 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291741 2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:23:58.296081 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291744 2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:23:58.296081 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291746 2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:23:58.296081 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291749 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:23:58.296081 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291752 2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:23:58.296081 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291754 2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:23:58.296081 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291757 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:23:58.296081 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291759 2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:23:58.296081 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291762 2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:23:58.296081 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291764 2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:23:58.296081 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291766 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:23:58.296081 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291769 2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:23:58.296081 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291772 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:23:58.296579 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291774 2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:23:58.296579 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291777 2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:23:58.296579 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291780 2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:23:58.296579 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291784 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:23:58.296579 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291787 2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:23:58.296579 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291789 2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:23:58.296579 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291792 2567 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:23:58.296579 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291794 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:23:58.296579 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291798 2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:23:58.296579 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291804 2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:23:58.296579 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291808 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:23:58.296579 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291811 2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:23:58.296579 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291814 2567 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:23:58.296579 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291817 2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:23:58.296579 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291820 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:23:58.296579 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291823 2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:23:58.296579 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291826 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:23:58.296579 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291829 2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:23:58.296579 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291831 2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:23:58.297047 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291835 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:23:58.297047 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291837 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:23:58.297047 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291840 2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:23:58.297047 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291845 2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:23:58.297047 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291847 2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:23:58.297047 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291850 2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:23:58.297047 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291852 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:23:58.297047 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291855 2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:23:58.297047 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291857 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:23:58.297047 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291860 2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:23:58.297047 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291863 2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:23:58.297047 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291866 2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:23:58.297047 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291869 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:23:58.297047 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291873 2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:23:58.297047 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291876 2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:23:58.297047 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291879 2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:23:58.297047 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291882 2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:23:58.297047 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291885 2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:23:58.297047 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291888 2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:23:58.297505 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291891 2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:23:58.297505 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291893 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:23:58.297505 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291896 2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:23:58.297505 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291900 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:23:58.297505 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291902 2567 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:23:58.297505 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291905 2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:23:58.297505 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291907 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:23:58.297505 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291910 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:23:58.297505 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291912 2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:23:58.297505 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291915 2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:23:58.297505 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291917 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:23:58.297505 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291920 2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:23:58.297505 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291922 2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:23:58.297505 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291925 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:23:58.297505 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291928 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:23:58.297505 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291930 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:23:58.297505 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291934 2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:23:58.297505 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291937 2567 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:23:58.297505 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291939 2567 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:23:58.297505 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291942 2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291944 2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291947 2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291949 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291953 2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.291955 2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.292934 2567 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.292945 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.293014 2567 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.293022 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294404 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294414 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294423 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294460 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294568 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294576 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294582 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294586 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294589 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294592 2567 flags.go:64] FLAG: --cgroup-root="" Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294595 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294599 2567 flags.go:64] FLAG: --client-ca-file="" Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294602 2567 flags.go:64] FLAG: --cloud-provider="external" Nov 02 01:23:58.298007 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294605 2567 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294612 2567 flags.go:64] FLAG: --cluster-domain="" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294615 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294618 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294621 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294625 2567 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294630 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294633 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294636 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294640 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294643 2567 flags.go:64] FLAG: --contention-profiling="false" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294646 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294649 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294652 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294655 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294660 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294663 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294666 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294669 2567 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294673 2567 flags.go:64] FLAG: --enable-server="true" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294676 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294683 2567 flags.go:64] FLAG: --event-burst="100" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294687 2567 flags.go:64] FLAG: --event-qps="50" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294690 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 01:23:58.298562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294694 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294697 2567 flags.go:64] FLAG: --eviction-hard="" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294702 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294705 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294707 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294711 2567 flags.go:64] FLAG: --eviction-soft="" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294714 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294717 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294721 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294724 2567 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294727 2567 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294730 2567 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294733 2567 flags.go:64] FLAG: --feature-gates="" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294737 2567 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294740 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294743 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294747 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294750 2567 flags.go:64] FLAG: --healthz-port="10248" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294753 2567 flags.go:64] FLAG: --help="false" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294756 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294759 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294762 2567 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294765 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 01:23:58.299149 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294769 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294772 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294775 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294778 2567 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294781 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294784 2567 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294788 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294791 2567 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294795 2567 flags.go:64] FLAG: --kube-reserved="" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294798 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294801 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294804 2567 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294807 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294809 2567 flags.go:64] FLAG: --lock-file="" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294812 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294815 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294818 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294825 2567 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294828 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294831 2567 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294834 2567 flags.go:64] FLAG: --logging-format="text" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294837 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294840 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294843 2567 flags.go:64] FLAG: --manifest-url="" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294846 2567 flags.go:64] FLAG: --manifest-url-header="" Nov 02 01:23:58.299709 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294852 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294855 2567 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294859 2567 flags.go:64] FLAG: --max-pods="110" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294862 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294865 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294868 2567 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294871 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294874 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294877 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294879 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294888 2567 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294891 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294895 2567 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294898 2567 flags.go:64] FLAG: --pod-cidr="" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294902 2567 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294905 2567 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294908 2567 flags.go:64] FLAG: --pods-per-core="0" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294911 2567 flags.go:64] FLAG: --port="10250" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294914 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294917 2567 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-071a45d511e7feffe" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294920 2567 flags.go:64] FLAG: --qos-reserved="" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294923 2567 flags.go:64] FLAG: --read-only-port="10255" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294926 2567 flags.go:64] FLAG: --register-node="true" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294929 2567 flags.go:64] FLAG: --register-with-taints="" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294933 2567 flags.go:64] FLAG: --registry-burst="10" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294936 2567 flags.go:64] FLAG: --registry-qps="5" Nov 02 01:23:58.300333 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294939 2567 flags.go:64] FLAG: --reserved-cpus="" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294942 2567 flags.go:64] FLAG: --reserved-memory="" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294946 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294949 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294952 2567 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294955 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294958 2567 flags.go:64] FLAG: --runonce="false" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294961 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294965 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294967 2567 flags.go:64] FLAG: --seccomp-default="false" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294970 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294973 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294976 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294979 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294983 2567 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294985 2567 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294988 2567 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294991 2567 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294995 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.294998 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.295001 2567 flags.go:64] FLAG: --system-cgroups="" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.295005 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.295012 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.295015 2567 flags.go:64] FLAG: --tls-cert-file="" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.295017 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 01:23:58.300985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.295021 2567 flags.go:64] FLAG: --tls-min-version="" Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.295024 2567 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.295027 2567 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.295030 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.295033 2567 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.295036 2567 flags.go:64] FLAG: --v="2" Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.295040 2567 flags.go:64] FLAG: --version="false" Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.295045 2567 flags.go:64] FLAG: --vmodule="" Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.295049 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.295052 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295192 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295196 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295199 2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295202 2567 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295205 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295208 2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295210 2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295213 2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295216 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295220 2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295224 2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:23:58.301628 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295227 2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:23:58.302127 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295230 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:23:58.302127 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295232 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:23:58.302127 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295235 2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:23:58.302127 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295238 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:23:58.302127 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295241 2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:23:58.302127 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295244 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:23:58.302127 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295246 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:23:58.302127 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295249 2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:23:58.302127 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295252 2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:23:58.302127 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295254 2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:23:58.302127 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295257 2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:23:58.302127 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295259 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:23:58.302127 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295262 2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:23:58.302127 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295264 2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:23:58.302127 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295267 2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:23:58.302127 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295269 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:23:58.302127 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295272 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:23:58.302127 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295274 2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:23:58.302127 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295277 2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:23:58.302127 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295280 2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:23:58.302629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295282 2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:23:58.302629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295285 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:23:58.302629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295288 2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:23:58.302629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295291 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:23:58.302629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295293 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:23:58.302629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295296 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:23:58.302629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295298 2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:23:58.302629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295300 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:23:58.302629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295303 2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:23:58.302629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295305 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:23:58.302629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295308 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:23:58.302629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295310 2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:23:58.302629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295313 2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:23:58.302629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295316 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:23:58.302629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295318 2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:23:58.302629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295321 2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:23:58.302629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295323 2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:23:58.302629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295326 2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:23:58.302629 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295328 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:23:58.303076 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295331 2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:23:58.303076 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295334 2567 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:23:58.303076 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295337 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:23:58.303076 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295340 2567 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:23:58.303076 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295342 2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:23:58.303076 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295345 2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:23:58.303076 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295347 2567 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:23:58.303076 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295350 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:23:58.303076 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295353 2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:23:58.303076 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295355 2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:23:58.303076 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295358 2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:23:58.303076 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295360 2567 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:23:58.303076 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295363 2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:23:58.303076 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295365 2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:23:58.303076 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295368 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:23:58.303076 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295370 2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:23:58.303076 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295373 2567 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:23:58.303076 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295376 2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:23:58.303076 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295380 2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:23:58.303601 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295385 2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:23:58.303601 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295388 2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:23:58.303601 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295391 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:23:58.303601 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295394 2567 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:23:58.303601 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295396 2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:23:58.303601 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295399 2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:23:58.303601 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295401 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:23:58.303601 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295404 2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:23:58.303601 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295406 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:23:58.303601 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295409 2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:23:58.303601 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295411 2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:23:58.303601 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295417 2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:23:58.303601 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295419 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:23:58.303601 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295422 2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:23:58.303601 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295424 2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:23:58.303601 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295427 2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:23:58.303601 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295430 2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:23:58.303601 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295432 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:23:58.303601 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295448 2567 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:23:58.304065 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295451 2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:23:58.304065 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295454 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:23:58.304065 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295457 2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:23:58.304065 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295459 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:23:58.304065 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295462 2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:23:58.304065 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295465 2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:23:58.304065 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295467 2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:23:58.304065 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295470 2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:23:58.304065 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295472 2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:23:58.304065 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295475 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:23:58.304065 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295477 2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:23:58.304065 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295481 2567 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:23:58.304065 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295486 2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:23:58.304065 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295489 2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:23:58.304065 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295491 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:23:58.304065 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295494 2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:23:58.304065 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295497 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:23:58.304484 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.295526 2567 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 01:23:58.304484 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295564 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:23:58.304484 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295570 2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:23:58.304484 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295573 2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:23:58.304484 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295576 2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:23:58.304484 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295579 2567 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:23:58.304484 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295581 2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:23:58.304484 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295585 2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:23:58.304484 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295588 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:23:58.304484 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295591 2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:23:58.304484 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295593 2567 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:23:58.304484 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295596 2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:23:58.304484 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295600 2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:23:58.304484 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295605 2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:23:58.304484 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295608 2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:23:58.304830 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295611 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:23:58.304830 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295614 2567 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:23:58.304830 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295616 2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:23:58.304830 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295619 2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:23:58.304830 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295621 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:23:58.304830 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295624 2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:23:58.304830 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295627 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:23:58.304830 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295629 2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:23:58.304830 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295632 2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:23:58.304830 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295635 2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:23:58.304830 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295638 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:23:58.304830 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295640 2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:23:58.304830 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295644 2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:23:58.304830 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295647 2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:23:58.304830 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295650 2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:23:58.304830 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295653 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:23:58.304830 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295656 2567 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:23:58.304830 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295658 2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:23:58.304830 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295661 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:23:58.304830 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295663 2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:23:58.305302 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295666 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:23:58.305302 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295669 2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:23:58.305302 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295672 2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:23:58.305302 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295674 2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:23:58.305302 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295678 2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:23:58.305302 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295680 2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:23:58.305302 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295683 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:23:58.305302 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295685 2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:23:58.305302 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295688 2567 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:23:58.305302 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295690 2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:23:58.305302 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295693 2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:23:58.305302 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295696 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:23:58.305302 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295698 2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:23:58.305302 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295701 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:23:58.305302 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295703 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:23:58.305302 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295706 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:23:58.305302 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295708 2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:23:58.305302 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295711 2567 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:23:58.305302 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295714 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:23:58.305302 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295716 2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:23:58.305791 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295719 2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:23:58.305791 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295721 2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:23:58.305791 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295724 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:23:58.305791 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295728 2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:23:58.305791 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295731 2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:23:58.305791 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295735 2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:23:58.305791 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295738 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:23:58.305791 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295741 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:23:58.305791 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295744 2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:23:58.305791 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295746 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:23:58.305791 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295749 2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:23:58.305791 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295751 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:23:58.305791 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295754 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:23:58.305791 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295757 2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:23:58.305791 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295759 2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:23:58.305791 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295761 2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:23:58.305791 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295765 2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:23:58.305791 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295768 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:23:58.305791 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295770 2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:23:58.306239 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295773 2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:23:58.306239 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295775 2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:23:58.306239 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295778 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:23:58.306239 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295781 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:23:58.306239 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295783 2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:23:58.306239 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295786 2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:23:58.306239 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295788 2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:23:58.306239 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295791 2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:23:58.306239 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295793 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:23:58.306239 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295796 2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:23:58.306239 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295799 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:23:58.306239 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295801 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:23:58.306239 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295804 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:23:58.306239 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295806 2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:23:58.306239 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295809 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:23:58.306239 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295811 2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:23:58.306239 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295814 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:23:58.306239 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295816 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:23:58.306239 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295820 2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:23:58.306239 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295824 2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:23:58.306759 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295826 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:23:58.306759 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295829 2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:23:58.306759 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295831 2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:23:58.306759 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295834 2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:23:58.306759 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295836 2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:23:58.306759 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295839 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:23:58.306759 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295842 2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:23:58.306759 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295845 2567 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:23:58.306759 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295848 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:23:58.306759 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295851 2567 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:23:58.306759 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295854 2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:23:58.306759 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295856 2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:23:58.306759 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.295859 2567 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:23:58.306759 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.302850 2567 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 01:23:58.306759 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.302867 2567 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 01:23:58.306759 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.302883 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 01:23:58.306759 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.302888 2567 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 01:23:58.306759 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.302944 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:23:58.306759 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.302950 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:23:58.307203 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.302953 2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:23:58.307203 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.302957 2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:23:58.307203 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.302959 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:23:58.307203 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.302962 2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:23:58.307203 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.302965 2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:23:58.307203 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.302968 2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:23:58.307203 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.302970 2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:23:58.307203 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.302973 2567 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:23:58.307203 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.302975 2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:23:58.307203 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.302978 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:23:58.307203 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.302980 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:23:58.307203 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.302983 2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:23:58.307203 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.302987 2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:23:58.307203 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.302991 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:23:58.307203 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.302994 2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:23:58.307203 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.302996 2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:23:58.307203 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.302999 2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:23:58.307203 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303002 2567 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:23:58.307203 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303005 2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:23:58.307658 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303007 2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:23:58.307658 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303010 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:23:58.307658 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303013 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:23:58.307658 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303015 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:23:58.307658 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303018 2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:23:58.307658 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303020 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:23:58.307658 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303023 2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:23:58.307658 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303026 2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:23:58.307658 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303031 2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:23:58.307658 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303034 2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:23:58.307658 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303037 2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:23:58.307658 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303039 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:23:58.307658 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303042 2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:23:58.307658 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303044 2567 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:23:58.307658 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303047 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:23:58.307658 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303049 2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:23:58.307658 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303052 2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:23:58.307658 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303055 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:23:58.307658 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303058 2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:23:58.307658 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303060 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:23:58.308131 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303063 2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:23:58.308131 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303065 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:23:58.308131 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303068 2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:23:58.308131 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303071 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:23:58.308131 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303073 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:23:58.308131 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303076 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:23:58.308131 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303078 2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:23:58.308131 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303081 2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:23:58.308131 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303084 2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:23:58.308131 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303086 2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:23:58.308131 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303089 2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:23:58.308131 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303092 2567 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:23:58.308131 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303094 2567 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:23:58.308131 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303097 2567 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:23:58.308131 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303099 2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:23:58.308131 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303102 2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:23:58.308131 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303104 2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:23:58.308131 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303107 2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:23:58.308131 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303109 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:23:58.308131 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303112 2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:23:58.308716 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303114 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:23:58.308716 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303118 2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:23:58.308716 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303120 2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:23:58.308716 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303123 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:23:58.308716 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303125 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:23:58.308716 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303128 2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:23:58.308716 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303130 2567 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:23:58.308716 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303133 2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:23:58.308716 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303136 2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:23:58.308716 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303138 2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:23:58.308716 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303141 2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:23:58.308716 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303143 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:23:58.308716 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303147 2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:23:58.308716 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303151 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:23:58.308716 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303154 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:23:58.308716 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303157 2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:23:58.308716 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303160 2567 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:23:58.308716 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303163 2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:23:58.308716 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303165 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:23:58.308716 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303168 2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:23:58.309200 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303171 2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:23:58.309200 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303173 2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:23:58.309200 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303176 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:23:58.309200 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303179 2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:23:58.309200 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303181 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:23:58.309200 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303184 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:23:58.309200 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303186 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:23:58.309200 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303189 2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:23:58.309200 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303191 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:23:58.309200 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303193 2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:23:58.309200 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303196 2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:23:58.309200 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303198 2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:23:58.309200 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303201 2567 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:23:58.309200 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303204 2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:23:58.309200 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303207 2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:23:58.309200 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303210 2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:23:58.309200 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303213 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:23:58.309200 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303215 2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:23:58.309200 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303217 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:23:58.309657 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303220 2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:23:58.309657 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303223 2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:23:58.309657 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303225 2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:23:58.309657 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303228 2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:23:58.309657 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303230 2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:23:58.309657 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303233 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:23:58.309657 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.303260 2567 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 01:23:58.309657 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303398 2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:23:58.309657 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303403 2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:23:58.309657 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303406 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:23:58.309657 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303409 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:23:58.309657 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303412 2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:23:58.309657 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303414 2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:23:58.309657 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303417 2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:23:58.309657 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303419 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:23:58.309657 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303422 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:23:58.310015 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303425 2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:23:58.310015 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303427 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:23:58.310015 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303430 2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:23:58.310015 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303447 2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:23:58.310015 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303453 2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:23:58.310015 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303456 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:23:58.310015 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303458 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:23:58.310015 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303461 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:23:58.310015 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303464 2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:23:58.310015 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303466 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:23:58.310015 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303469 2567 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:23:58.310015 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303472 2567 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:23:58.310015 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303475 2567 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:23:58.310015 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303478 2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:23:58.310015 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303480 2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:23:58.310015 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303483 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:23:58.310015 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303486 2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:23:58.310015 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303489 2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:23:58.310015 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303491 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:23:58.310458 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303494 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:23:58.310458 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303497 2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:23:58.310458 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303499 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:23:58.310458 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303502 2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:23:58.310458 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303505 2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:23:58.310458 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303507 2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:23:58.310458 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303510 2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:23:58.310458 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303512 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:23:58.310458 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303515 2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:23:58.310458 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303517 2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:23:58.310458 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303520 2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:23:58.310458 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303523 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:23:58.310458 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303525 2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:23:58.310458 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303528 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:23:58.310458 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303530 2567 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:23:58.310458 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303533 2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:23:58.310458 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303535 2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:23:58.310458 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303538 2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:23:58.310458 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303540 2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:23:58.310458 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303543 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:23:58.310930 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303547 2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:23:58.310930 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303550 2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:23:58.310930 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303552 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:23:58.310930 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303555 2567 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:23:58.310930 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303557 2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:23:58.310930 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303559 2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:23:58.310930 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303563 2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:23:58.310930 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303565 2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:23:58.310930 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303568 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:23:58.310930 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303570 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:23:58.310930 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303572 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:23:58.310930 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303575 2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:23:58.310930 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303577 2567 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:23:58.310930 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303580 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:23:58.310930 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303582 2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:23:58.310930 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303585 2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:23:58.310930 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303588 2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:23:58.310930 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303590 2567 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:23:58.310930 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303593 2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:23:58.310930 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303596 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:23:58.311445 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303598 2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:23:58.311445 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303601 2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:23:58.311445 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303603 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:23:58.311445 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303606 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:23:58.311445 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303609 2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:23:58.311445 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303611 2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:23:58.311445 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303614 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:23:58.311445 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303617 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:23:58.311445 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303619 2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:23:58.311445 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303622 2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:23:58.311445 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303624 2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:23:58.311445 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303628 2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:23:58.311445 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303634 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:23:58.311445 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303637 2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:23:58.311445 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303640 2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:23:58.311445 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303643 2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:23:58.311445 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303646 2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:23:58.311445 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303648 2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:23:58.311445 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303651 2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:23:58.311891 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303654 2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:23:58.311891 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303657 2567 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:23:58.311891 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303659 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:23:58.311891 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303662 2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:23:58.311891 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303664 2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:23:58.311891 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303667 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:23:58.311891 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303669 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:23:58.311891 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303672 2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:23:58.311891 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303674 2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:23:58.311891 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303677 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:23:58.311891 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303679 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:23:58.311891 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303682 2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:23:58.311891 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303684 2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:23:58.311891 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303686 2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:23:58.311891 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303689 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:23:58.311891 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303691 2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:23:58.311891 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303694 2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:23:58.311891 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303697 2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:23:58.311891 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303699 2567 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:23:58.312341 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.303722 2567 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 01:23:58.312341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303755 2567 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:23:58.312341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303758 2567 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:23:58.312341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303761 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:23:58.312341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303764 2567 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:23:58.312341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303766 2567 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:23:58.312341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303770 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:23:58.312341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303773 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:23:58.312341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303776 2567 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:23:58.312341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303778 2567 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:23:58.312341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303781 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:23:58.312341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303783 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:23:58.312341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303786 2567 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:23:58.312341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303788 2567 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:23:58.312341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303791 2567 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:23:58.312341 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303794 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:23:58.312773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303796 2567 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:23:58.312773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303799 2567 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:23:58.312773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303802 2567 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:23:58.312773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303804 2567 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:23:58.312773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303807 2567 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:23:58.312773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303809 2567 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:23:58.312773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303811 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:23:58.312773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303814 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:23:58.312773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303816 2567 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:23:58.312773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303819 2567 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:23:58.312773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303821 2567 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:23:58.312773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303824 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:23:58.312773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303827 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:23:58.312773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303829 2567 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:23:58.312773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303832 2567 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:23:58.312773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303834 2567 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:23:58.312773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303838 2567 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:23:58.312773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303842 2567 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:23:58.312773 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303845 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:23:58.313235 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303847 2567 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:23:58.313235 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303850 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:23:58.313235 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303853 2567 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:23:58.313235 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303856 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:23:58.313235 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303859 2567 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:23:58.313235 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303861 2567 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:23:58.313235 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303864 2567 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:23:58.313235 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303866 2567 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:23:58.313235 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303869 2567 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:23:58.313235 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303872 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:23:58.313235 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303874 2567 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:23:58.313235 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303877 2567 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:23:58.313235 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303880 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:23:58.313235 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303882 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:23:58.313235 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303885 2567 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:23:58.313235 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303888 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:23:58.313235 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303890 2567 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:23:58.313235 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303893 2567 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:23:58.313235 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303896 2567 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:23:58.313235 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303899 2567 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:23:58.313699 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303901 2567 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:23:58.313699 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303903 2567 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:23:58.313699 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303906 2567 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:23:58.313699 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303909 2567 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:23:58.313699 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303911 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:23:58.313699 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303914 2567 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:23:58.313699 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303916 2567 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:23:58.313699 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303919 2567 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:23:58.313699 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303921 2567 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:23:58.313699 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303924 2567 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:23:58.313699 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303926 2567 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:23:58.313699 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303929 2567 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:23:58.313699 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303931 2567 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:23:58.313699 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303934 2567 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:23:58.313699 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303937 2567 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:23:58.313699 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303942 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:23:58.313699 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303944 2567 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:23:58.313699 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303947 2567 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:23:58.313699 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303949 2567 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:23:58.313699 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303952 2567 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:23:58.314232 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303954 2567 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:23:58.314232 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303957 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:23:58.314232 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303959 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:23:58.314232 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303961 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:23:58.314232 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303964 2567 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:23:58.314232 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303967 2567 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:23:58.314232 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303969 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:23:58.314232 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303972 2567 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:23:58.314232 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303975 2567 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:23:58.314232 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303977 2567 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:23:58.314232 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303980 2567 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:23:58.314232 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303982 2567 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:23:58.314232 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303985 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:23:58.314232 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303987 2567 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:23:58.314232 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303989 2567 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:23:58.314232 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303994 2567 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:23:58.314232 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303996 2567 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:23:58.314232 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.303999 2567 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:23:58.314232 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.304001 2567 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:23:58.314760 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.304004 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:23:58.314760 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.304006 2567 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:23:58.314760 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.304009 2567 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:23:58.314760 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.304011 2567 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:23:58.314760 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.304014 2567 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:23:58.314760 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.304025 2567 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:23:58.314760 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.304030 2567 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:23:58.314760 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.304033 2567 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:23:58.314760 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.304038 2567 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:23:58.314760 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.304041 2567 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:23:58.314760 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.304043 2567 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:23:58.314760 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.304046 2567 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:23:58.314760 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:23:58.304049 2567 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:23:58.314760 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.304104 2567 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 01:23:58.314760 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.305908 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 01:23:58.314760 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.306963 2567 server.go:1014] "Starting client certificate rotation" Nov 02 01:23:58.314760 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.307056 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 01:23:58.314760 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.307081 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 01:23:58.315202 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.308894 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 01:23:58.315202 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.313706 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 01:23:58.322335 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.322316 2567 log.go:25] "Validated CRI v1 runtime API" Nov 02 01:23:58.326281 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.326265 2567 log.go:25] "Validated CRI v1 image API" Nov 02 01:23:58.327064 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.327040 2567 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 01:23:58.329424 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.329404 2567 fs.go:135] Filesystem UUIDs: map[488f296c-96e6-49a6-af0f-103489929e55:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d9515f3b-97e5-460a-9d3f-0f82133e4c90:/dev/nvme0n1p3] Nov 02 01:23:58.329522 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.329422 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 01:23:58.335181 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.335074 2567 manager.go:217] Machine: {Timestamp:2026-11-02 01:23:58.333293597 +0000 UTC m=+0.321934542 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108852 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29d8484f17f5e354463354903f2582 SystemUUID:ec29d848-4f17-f5e3-5446-3354903f2582 BootID:40e3b1ce-4f70-40d8-9354-27758804a79e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:64:d9:a7:c9:d1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:64:d9:a7:c9:d1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:31:7c:01:c8:4f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 01:23:58.335795 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.335785 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 01:23:58.335875 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.335864 2567 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 01:23:58.339535 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.339506 2567 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 01:23:58.339694 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.339537 2567 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-6.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 01:23:58.339773 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.339700 2567 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 01:23:58.339773 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.339713 2567 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 01:23:58.339773 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.339750 2567 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 01:23:58.339773 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.339769 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 01:23:58.340858 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.340845 2567 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 01:23:58.343696 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.343682 2567 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 01:23:58.343828 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.343817 2567 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 01:23:58.344583 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.344572 2567 kubelet.go:493] "Attempting to sync node with API server" Nov 02 01:23:58.344635 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.344590 2567 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 01:23:58.344635 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.344606 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 01:23:58.344635 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.344621 2567 kubelet.go:405] "Adding apiserver pod source" Nov 02 01:23:58.344635 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.344633 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 01:23:58.344991 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.344976 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 01:23:58.347073 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.347056 2567 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 01:23:58.351044 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.351031 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 01:23:58.351775 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.351763 2567 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 01:23:58.351811 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.351798 2567 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 01:23:58.351957 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.351947 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 01:23:58.351987 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.351963 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 01:23:58.351987 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.351970 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 01:23:58.351987 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.351976 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 01:23:58.352067 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.351989 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 01:23:58.352067 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.351998 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 01:23:58.352482 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.352472 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 01:23:58.352518 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.352484 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 01:23:58.352518 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.352491 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 01:23:58.352518 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.352497 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 01:23:58.352518 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.352510 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 01:23:58.352518 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.352518 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 01:23:58.353557 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.353548 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 01:23:58.353557 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.353557 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 01:23:58.357160 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.357146 2567 server.go:1263] "Started kubelet" Nov 02 01:23:58.357242 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.357211 2567 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 01:23:58.357305 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.357253 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 01:23:58.357339 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.357328 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 01:23:58.357827 ip-10-0-129-6 systemd[1]: Started Kubernetes Kubelet. Nov 02 01:23:58.358880 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.358313 2567 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 01:23:58.359039 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.359025 2567 server.go:317] "Adding debug handlers to kubelet server" Nov 02 01:23:58.363196 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:23:58.363176 2567 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 01:23:58.363582 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.363566 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 01:23:58.364536 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.364521 2567 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 01:23:58.364635 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:23:58.364616 2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-6.us-west-1.compute.internal\" not found" Nov 02 01:23:58.365387 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.365252 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 01:23:58.365565 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.365306 2567 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 01:23:58.365565 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.365565 2567 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 01:23:58.365728 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.365613 2567 reconstruct.go:97] "Volume reconstruction finished" Nov 02 01:23:58.365728 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.365623 2567 reconciler.go:29] "Reconciler: start to sync state" Nov 02 01:23:58.367002 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.366979 2567 factory.go:55] Registering systemd factory Nov 02 01:23:58.367002 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.367005 2567 factory.go:223] Registration of the systemd container factory successfully Nov 02 01:23:58.367337 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.367321 2567 factory.go:153] Registering CRI-O factory Nov 02 01:23:58.367404 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.367341 2567 factory.go:223] Registration of the crio container factory successfully Nov 02 01:23:58.367404 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.367389 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 01:23:58.367504 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.367409 2567 factory.go:103] Registering Raw factory Nov 02 01:23:58.367504 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.367421 2567 manager.go:1199] Started watching for new ooms in manager Nov 02 01:23:58.369293 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.369278 2567 manager.go:319] Starting recovery of all containers Nov 02 01:23:58.379873 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.379858 2567 manager.go:324] Recovery completed Nov 02 01:23:58.384433 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.384419 2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:23:58.386175 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.386153 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:23:58.386242 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.386194 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:23:58.386242 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.386210 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 01:23:58.386887 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.386847 2567 cpu_manager.go:226] "Starting" policy="none" Nov 02 01:23:58.386887 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.386856 2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:23:58.386887 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.386860 2567 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 01:23:58.387069 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.386890 2567 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 01:23:58.387725 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.387711 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:23:58.387794 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.387737 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:23:58.387794 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.387747 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 01:23:58.388078 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.388066 2567 policy_none.go:50] "Start" Nov 02 01:23:58.388120 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.388083 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 01:23:58.388120 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.388093 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 01:23:58.389152 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.389141 2567 policy_none.go:44] "Start" Nov 02 01:23:58.432288 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.420049 2567 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 01:23:58.432288 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.427892 2567 manager.go:341] "Starting Device Plugin manager" Nov 02 01:23:58.432288 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:23:58.427924 2567 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 01:23:58.432288 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.427936 2567 server.go:87] "Starting device plugin registration server" Nov 02 01:23:58.432288 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.428595 2567 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 01:23:58.432288 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.428610 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 01:23:58.432288 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.428722 2567 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 01:23:58.432288 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.428794 2567 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 01:23:58.432288 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.428801 2567 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 01:23:58.432288 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:23:58.429387 2567 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 01:23:58.432288 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:23:58.429456 2567 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-6.us-west-1.compute.internal\" not found" Nov 02 01:23:58.432288 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.430186 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 01:23:58.432288 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.431422 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 01:23:58.432288 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.431464 2567 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 01:23:58.432288 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.431490 2567 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 01:23:58.432288 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:23:58.431535 2567 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 01:23:58.510300 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:23:58.510255 2567 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-6.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 01:23:58.510617 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.510556 2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-6.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 01:23:58.513573 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:23:58.510253 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-6.us-west-1.compute.internal.18e415e822958769 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-6.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-6.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 01:23:58.357120873 +0000 UTC m=+0.345761817,LastTimestamp:2026-11-02 01:23:58.357120873 +0000 UTC m=+0.345761817,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-6.us-west-1.compute.internal,}" Nov 02 01:23:58.529151 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.529128 2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:23:58.529989 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.529965 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:23:58.530076 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.530003 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:23:58.530076 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.530020 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 01:23:58.530076 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.530045 2567 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.532640 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.532612 2567 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal"] Nov 02 01:23:58.532697 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.532688 2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:23:58.533337 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.533324 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:23:58.533399 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.533351 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:23:58.533399 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.533367 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 01:23:58.534610 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.534579 2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:23:58.535187 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.535174 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.535241 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.535203 2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:23:58.536804 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.536785 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:23:58.536875 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.536805 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:23:58.536875 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.536820 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:23:58.536875 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.536829 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:23:58.536875 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.536834 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 01:23:58.536875 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.536838 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 01:23:58.537981 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.537966 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.537981 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.537991 2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:23:58.543055 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.543040 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:23:58.543135 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.543066 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:23:58.543135 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.543083 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 01:23:58.562657 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:23:58.562633 2567 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-6.us-west-1.compute.internal\" not found" node="ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.566703 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.566682 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/817dd2f8fadd21a6e3346c963f0534ec-config\") pod \"kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal\" (UID: \"817dd2f8fadd21a6e3346c963f0534ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.566785 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.566708 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c52cca1e001b790fd8c346cdd80a9e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal\" (UID: \"1c52cca1e001b790fd8c346cdd80a9e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.566785 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.566727 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c52cca1e001b790fd8c346cdd80a9e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal\" (UID: \"1c52cca1e001b790fd8c346cdd80a9e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.566785 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:23:58.566769 2567 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-6.us-west-1.compute.internal\" not found" node="ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.593158 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:23:58.593121 2567 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 02 01:23:58.593158 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:23:58.593119 2567 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 01:23:58.593300 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:23:58.593104 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-6.us-west-1.compute.internal.18e415e82450dfbf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-6.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-6.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-6.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 01:23:58.386175935 +0000 UTC m=+0.374816883,LastTimestamp:2026-11-02 01:23:58.386175935 +0000 UTC m=+0.374816883,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-6.us-west-1.compute.internal,}" Nov 02 01:23:58.593368 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:23:58.593294 2567 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 01:23:58.593478 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:23:58.593457 2567 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-6.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 01:23:58.594332 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.594315 2567 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 01:23:58.594395 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.594381 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hg9lf" Nov 02 01:23:58.667212 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.667190 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/817dd2f8fadd21a6e3346c963f0534ec-config\") pod \"kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal\" (UID: \"817dd2f8fadd21a6e3346c963f0534ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.667312 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.667217 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c52cca1e001b790fd8c346cdd80a9e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal\" (UID: \"1c52cca1e001b790fd8c346cdd80a9e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.667312 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.667247 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c52cca1e001b790fd8c346cdd80a9e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal\" (UID: \"1c52cca1e001b790fd8c346cdd80a9e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.667312 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.667301 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c52cca1e001b790fd8c346cdd80a9e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal\" (UID: \"1c52cca1e001b790fd8c346cdd80a9e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.667424 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.667303 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/817dd2f8fadd21a6e3346c963f0534ec-config\") pod \"kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal\" (UID: \"817dd2f8fadd21a6e3346c963f0534ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.667424 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.667304 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c52cca1e001b790fd8c346cdd80a9e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal\" (UID: \"1c52cca1e001b790fd8c346cdd80a9e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.673809 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:23:58.673792 2567 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-129-6.us-west-1.compute.internal\" not found" node="ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.816433 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:23:58.816379 2567 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-6.us-west-1.compute.internal\" not found" node="ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.865613 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.865594 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.870895 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.870881 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.874002 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.873989 2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:23:58.875060 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.875045 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:23:58.875102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.875076 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:23:58.875102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.875090 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 01:23:58.875160 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.875117 2567 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.934178 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:23:58.934157 2567 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:23:58.934178 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:23:58.934180 2567 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-6.us-west-1.compute.internal\": node \"ip-10-0-129-6.us-west-1.compute.internal\" not found" Nov 02 01:23:59.049513 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:23:59.049486 2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-6.us-west-1.compute.internal\" not found" Nov 02 01:23:59.149653 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:23:59.149587 2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-6.us-west-1.compute.internal\" not found" Nov 02 01:24:01.076766 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:01.076741 2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-6.us-west-1.compute.internal\" not found" Nov 02 01:24:01.133753 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:01.133730 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 01:24:01.134133 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:01.133868 2567 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a379cc44211674ec397976e30fa324b1-fb94896701b40ffa.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-129-6.us-west-1.compute.internal.18e415e8245143c8\": read tcp 10.0.129.6:50084->3.133.195.157:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-129-6.us-west-1.compute.internal.18e415e8245143c8 default 4736 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-6.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-6.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-6.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 01:23:58 +0000 UTC,LastTimestamp:2026-11-02 01:23:58.533358536 +0000 UTC m=+0.521999482,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-6.us-west-1.compute.internal,}" Nov 02 01:24:01.177182 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:01.177161 2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-6.us-west-1.compute.internal\" not found" Nov 02 01:24:01.277627 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:01.277559 2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-6.us-west-1.compute.internal\" not found" Nov 02 01:24:01.378235 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:01.378201 2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-6.us-west-1.compute.internal\" not found" Nov 02 01:24:01.426174 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:01.426138 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 01:19:00 +0000 UTC" deadline="2028-03-28 12:51:30.414019995 +0000 UTC" Nov 02 01:24:01.426174 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:01.426169 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12299h27m28.987853792s" Nov 02 01:24:01.471605 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:01.471582 2567 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:24:01.472726 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:01.472709 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:24:01.472812 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:01.472739 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:24:01.472812 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:01.472749 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 01:24:01.478843 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:01.478827 2567 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-6.us-west-1.compute.internal\" not found" Nov 02 01:24:01.487943 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:01.487929 2567 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:24:01.489143 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:01.489128 2567 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:24:01.491229 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:01.491216 2567 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:01.554208 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:01.554147 2567 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 01:24:01.555390 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:01.555377 2567 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:01.616338 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:01.616307 2567 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 01:24:01.739573 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:01.739541 2567 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:24:01.760648 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:01.760599 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817dd2f8fadd21a6e3346c963f0534ec.slice/crio-60ee38920eb119bf15b637d23b79c11689392dd6adbec04062c5121447e03e03 WatchSource:0}: Error finding container 60ee38920eb119bf15b637d23b79c11689392dd6adbec04062c5121447e03e03: Status 404 returned error can't find the container with id 60ee38920eb119bf15b637d23b79c11689392dd6adbec04062c5121447e03e03 Nov 02 01:24:01.763271 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:01.763243 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c52cca1e001b790fd8c346cdd80a9e7.slice/crio-0a6d9848ff171086e151f397491a8f514dc8832592a239c33122c3284875456f WatchSource:0}: Error finding container 0a6d9848ff171086e151f397491a8f514dc8832592a239c33122c3284875456f: Status 404 returned error can't find the container with id 0a6d9848ff171086e151f397491a8f514dc8832592a239c33122c3284875456f Nov 02 01:24:01.767583 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:01.767567 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:24:01.796847 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:01.796825 2567 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:24:02.172064 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.172029 2567 apiserver.go:52] "Watching apiserver" Nov 02 01:24:02.191196 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.191173 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 01:24:02.261628 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.261575 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" event={"ID":"1c52cca1e001b790fd8c346cdd80a9e7","Type":"ContainerStarted","Data":"0a6d9848ff171086e151f397491a8f514dc8832592a239c33122c3284875456f"} Nov 02 01:24:02.262761 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.262738 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal" event={"ID":"817dd2f8fadd21a6e3346c963f0534ec","Type":"ContainerStarted","Data":"60ee38920eb119bf15b637d23b79c11689392dd6adbec04062c5121447e03e03"} Nov 02 01:24:02.276861 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.275508 2567 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 01:24:02.276861 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.276265 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl","openshift-cluster-node-tuning-operator/tuned-gwkc5","openshift-dns/node-resolver-ht26h","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal","kube-system/global-pull-secret-syncer-7szx6","openshift-image-registry/node-ca-rrf9f","openshift-ovn-kubernetes/ovnkube-node-qzhvs","kube-system/konnectivity-agent-7mk45","openshift-multus/multus-additional-cni-plugins-mgjgw","openshift-multus/multus-wzzsb","openshift-multus/network-metrics-daemon-flwvx","openshift-network-diagnostics/network-check-target-kgbsx","openshift-network-operator/iptables-alerter-zzm69"] Nov 02 01:24:02.278622 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.278585 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.279686 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.279654 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.280838 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.280824 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.281927 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.281906 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zzm69" Nov 02 01:24:02.282984 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.282968 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rrf9f" Nov 02 01:24:02.284114 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.284098 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.285454 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.285439 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:02.285530 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:02.285496 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flwvx" podUID="d628518e-2ab7-4acb-acf7-6fc96f359a64" Nov 02 01:24:02.286824 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.286810 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.288024 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.288010 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:02.288097 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:02.288059 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgbsx" podUID="85784876-9397-4bec-8ade-878a15c09f40" Nov 02 01:24:02.289278 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.289264 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7mk45" Nov 02 01:24:02.290538 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.290524 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:02.290632 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:02.290573 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7szx6" podUID="79f97db3-814b-4849-ab8d-305dbcd3449c" Nov 02 01:24:02.291919 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.291902 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ht26h" Nov 02 01:24:02.300508 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.300486 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79f97db3-814b-4849-ab8d-305dbcd3449c-dbus\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:02.300570 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.300524 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b91ad19-9236-4395-9e54-09470929bf35-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.300633 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.300590 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-var-lib-cni-bin\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.300691 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.300641 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-node-log\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.300691 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.300683 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bb2a916a-fc15-4bdc-b28a-c2473a543ce5-konnectivity-ca\") pod \"konnectivity-agent-7mk45\" (UID: \"bb2a916a-fc15-4bdc-b28a-c2473a543ce5\") " pod="kube-system/konnectivity-agent-7mk45" Nov 02 01:24:02.300779 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.300714 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:02.300779 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.300741 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-var-lib-kubelet\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.300841 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.300778 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-run-systemd\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.300841 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.300823 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-cni-netd\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.300898 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.300856 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-724nj\" (UniqueName: \"kubernetes.io/projected/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-kube-api-access-724nj\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.300929 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.300896 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-sysctl-d\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.300962 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.300940 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqppx\" (UniqueName: \"kubernetes.io/projected/9b91ad19-9236-4395-9e54-09470929bf35-kube-api-access-lqppx\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.301008 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.300966 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-cni-binary-copy\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.301008 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.300983 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-var-lib-cni-multus\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.301070 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301049 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b083c3f1-9c74-4313-a939-ff8aa5769c8d-ovn-node-metrics-cert\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.301111 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301082 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-sys-fs\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.301189 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301124 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-var-lib-kubelet\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.301189 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301150 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-run-netns\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.301189 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301167 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-hostroot\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.301282 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301221 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-run-openvswitch\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.301313 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301301 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.301346 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301325 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.301388 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301373 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1ed8a22-43d0-4ca3-9f7b-900231ad3685-hosts-file\") pod \"node-resolver-ht26h\" (UID: \"a1ed8a22-43d0-4ca3-9f7b-900231ad3685\") " pod="openshift-dns/node-resolver-ht26h" Nov 02 01:24:02.301426 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301413 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-modprobe-d\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.301460 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301437 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-kubernetes\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.301490 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301461 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-log-socket\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.301490 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301477 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e534da40-0733-4670-ba7a-839e0e4b1b4f-tmp\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.301547 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301492 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-run-k8s-cni-cncf-io\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.301547 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301535 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a128d9e6-7be3-4c73-88f3-ba6b9ea59971-host\") pod \"node-ca-rrf9f\" (UID: \"a128d9e6-7be3-4c73-88f3-ba6b9ea59971\") " pod="openshift-image-registry/node-ca-rrf9f" Nov 02 01:24:02.301602 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301568 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a128d9e6-7be3-4c73-88f3-ba6b9ea59971-serviceca\") pod \"node-ca-rrf9f\" (UID: \"a128d9e6-7be3-4c73-88f3-ba6b9ea59971\") " pod="openshift-image-registry/node-ca-rrf9f" Nov 02 01:24:02.301633 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301605 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j98n8\" (UniqueName: \"kubernetes.io/projected/a128d9e6-7be3-4c73-88f3-ba6b9ea59971-kube-api-access-j98n8\") pod \"node-ca-rrf9f\" (UID: \"a128d9e6-7be3-4c73-88f3-ba6b9ea59971\") " pod="openshift-image-registry/node-ca-rrf9f" Nov 02 01:24:02.301633 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301621 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b083c3f1-9c74-4313-a939-ff8aa5769c8d-env-overrides\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.301713 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301644 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-device-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.301713 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301664 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-sysctl-conf\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.301713 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301708 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-host\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.301806 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301736 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/673cb927-a5fc-4237-9b4c-fb196801d70c-host-slash\") pod \"iptables-alerter-zzm69\" (UID: \"673cb927-a5fc-4237-9b4c-fb196801d70c\") " pod="openshift-network-operator/iptables-alerter-zzm69" Nov 02 01:24:02.301806 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301764 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-system-cni-dir\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.301806 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301792 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-cni-dir\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.301902 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301810 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-var-lib-openvswitch\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.301902 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301840 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-socket-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.301902 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301861 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-lib-modules\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.301902 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301876 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b91ad19-9236-4395-9e54-09470929bf35-system-cni-dir\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.302017 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301904 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b91ad19-9236-4395-9e54-09470929bf35-cni-binary-copy\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.302017 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301938 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ed8a22-43d0-4ca3-9f7b-900231ad3685-tmp-dir\") pod \"node-resolver-ht26h\" (UID: \"a1ed8a22-43d0-4ca3-9f7b-900231ad3685\") " pod="openshift-dns/node-resolver-ht26h" Nov 02 01:24:02.302017 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.301977 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bb2a916a-fc15-4bdc-b28a-c2473a543ce5-agent-certs\") pod \"konnectivity-agent-7mk45\" (UID: \"bb2a916a-fc15-4bdc-b28a-c2473a543ce5\") " pod="kube-system/konnectivity-agent-7mk45" Nov 02 01:24:02.302017 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302010 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-slash\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.302169 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302051 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-run-ovn-kubernetes\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.302169 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302075 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/673cb927-a5fc-4237-9b4c-fb196801d70c-iptables-alerter-script\") pod \"iptables-alerter-zzm69\" (UID: \"673cb927-a5fc-4237-9b4c-fb196801d70c\") " pod="openshift-network-operator/iptables-alerter-zzm69" Nov 02 01:24:02.302169 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302111 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b91ad19-9236-4395-9e54-09470929bf35-cnibin\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.302169 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302138 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-systemd-units\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.302169 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302158 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-etc-selinux\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.302375 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302174 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b083c3f1-9c74-4313-a939-ff8aa5769c8d-ovnkube-config\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.302375 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302212 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-registration-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.302375 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302238 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-systemd\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.302375 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302272 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-run\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.302375 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302303 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-cnibin\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.302375 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302333 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-etc-kubernetes\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.302375 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302360 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b083c3f1-9c74-4313-a939-ff8aa5769c8d-ovnkube-script-lib\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.302620 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302399 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-sys\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.302620 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302415 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-daemon-config\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.302620 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302439 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-run-multus-certs\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.302620 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302456 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg9vq\" (UniqueName: \"kubernetes.io/projected/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-kube-api-access-jg9vq\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.302620 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302474 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-kubelet\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.302620 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302496 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-etc-openvswitch\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.302620 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302526 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-socket-dir-parent\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.302620 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302548 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-run-netns\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.302620 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302563 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-tuned\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.302620 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302591 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs\") pod \"network-metrics-daemon-flwvx\" (UID: \"d628518e-2ab7-4acb-acf7-6fc96f359a64\") " pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:02.302620 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302608 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dzj7\" (UniqueName: \"kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7\") pod \"network-check-target-kgbsx\" (UID: \"85784876-9397-4bec-8ade-878a15c09f40\") " pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:02.303042 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302636 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79f97db3-814b-4849-ab8d-305dbcd3449c-kubelet-config\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:02.303042 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302701 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b91ad19-9236-4395-9e54-09470929bf35-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.303042 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302733 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b91ad19-9236-4395-9e54-09470929bf35-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.303042 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302766 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-run-ovn\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.303042 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302810 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-cni-bin\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.303042 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302849 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrnrh\" (UniqueName: \"kubernetes.io/projected/673cb927-a5fc-4237-9b4c-fb196801d70c-kube-api-access-xrnrh\") pod \"iptables-alerter-zzm69\" (UID: \"673cb927-a5fc-4237-9b4c-fb196801d70c\") " pod="openshift-network-operator/iptables-alerter-zzm69" Nov 02 01:24:02.303042 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302912 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-conf-dir\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.303042 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302946 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crplh\" (UniqueName: \"kubernetes.io/projected/b083c3f1-9c74-4313-a939-ff8aa5769c8d-kube-api-access-crplh\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.303042 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.302993 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4wlx\" (UniqueName: \"kubernetes.io/projected/a1ed8a22-43d0-4ca3-9f7b-900231ad3685-kube-api-access-b4wlx\") pod \"node-resolver-ht26h\" (UID: \"a1ed8a22-43d0-4ca3-9f7b-900231ad3685\") " pod="openshift-dns/node-resolver-ht26h" Nov 02 01:24:02.303042 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.303026 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-sysconfig\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.303546 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.303069 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zjgc\" (UniqueName: \"kubernetes.io/projected/e534da40-0733-4670-ba7a-839e0e4b1b4f-kube-api-access-5zjgc\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.303546 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.303088 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh2r4\" (UniqueName: \"kubernetes.io/projected/d628518e-2ab7-4acb-acf7-6fc96f359a64-kube-api-access-rh2r4\") pod \"network-metrics-daemon-flwvx\" (UID: \"d628518e-2ab7-4acb-acf7-6fc96f359a64\") " pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:02.331023 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.331000 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 01:24:02.331023 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.331003 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 01:24:02.331179 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.331037 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 01:24:02.331179 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.331114 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 01:24:02.331179 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.331165 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 01:24:02.331343 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.331198 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 01:24:02.331343 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.331287 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4gwsn\"" Nov 02 01:24:02.331449 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.331352 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-gsg88\"" Nov 02 01:24:02.332331 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.332312 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:24:02.332487 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.332452 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p5zpq\"" Nov 02 01:24:02.332556 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.332507 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 01:24:02.375861 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.375833 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 01:24:02.375993 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.375869 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 01:24:02.376334 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.376313 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gh69v\"" Nov 02 01:24:02.376458 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.376444 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 01:24:02.376515 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.376495 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:24:02.376515 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.376519 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 01:24:02.376515 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.376872 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 01:24:02.376515 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.376882 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pw4q8\"" Nov 02 01:24:02.376515 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.376901 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 01:24:02.376515 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.377028 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 01:24:02.376515 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.377058 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 01:24:02.378740 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.378718 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 01:24:02.380242 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.380226 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 01:24:02.380308 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.380263 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 01:24:02.380308 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.380285 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wwr4r\"" Nov 02 01:24:02.380405 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.380326 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 01:24:02.380405 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.380324 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2dvkb\"" Nov 02 01:24:02.380405 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.380332 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kv2jl\"" Nov 02 01:24:02.380405 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.380349 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 01:24:02.380405 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.380381 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 01:24:02.380405 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.380401 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 01:24:02.380592 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.380431 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wfbzc\"" Nov 02 01:24:02.380592 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.380431 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 01:24:02.380592 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.380521 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 01:24:02.380592 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.380547 2567 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 01:24:02.380592 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.380572 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 01:24:02.380592 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.380588 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pv8gj" Nov 02 01:24:02.392168 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.392151 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 01:24:02.403255 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403237 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-var-lib-openvswitch\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.403311 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403265 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-socket-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.403311 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403282 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-lib-modules\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.403311 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403303 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b91ad19-9236-4395-9e54-09470929bf35-system-cni-dir\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.403411 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403341 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-var-lib-openvswitch\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.403411 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403359 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b91ad19-9236-4395-9e54-09470929bf35-cni-binary-copy\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.403411 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403384 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ed8a22-43d0-4ca3-9f7b-900231ad3685-tmp-dir\") pod \"node-resolver-ht26h\" (UID: \"a1ed8a22-43d0-4ca3-9f7b-900231ad3685\") " pod="openshift-dns/node-resolver-ht26h" Nov 02 01:24:02.403411 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403392 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b91ad19-9236-4395-9e54-09470929bf35-system-cni-dir\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.403595 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403411 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bb2a916a-fc15-4bdc-b28a-c2473a543ce5-agent-certs\") pod \"konnectivity-agent-7mk45\" (UID: \"bb2a916a-fc15-4bdc-b28a-c2473a543ce5\") " pod="kube-system/konnectivity-agent-7mk45" Nov 02 01:24:02.403595 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403435 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-slash\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.403595 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403451 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-run-ovn-kubernetes\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.403595 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403445 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-lib-modules\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.403595 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403466 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/673cb927-a5fc-4237-9b4c-fb196801d70c-iptables-alerter-script\") pod \"iptables-alerter-zzm69\" (UID: \"673cb927-a5fc-4237-9b4c-fb196801d70c\") " pod="openshift-network-operator/iptables-alerter-zzm69" Nov 02 01:24:02.403595 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403459 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-socket-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.403595 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403485 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b91ad19-9236-4395-9e54-09470929bf35-cnibin\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.403595 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403509 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-systemd-units\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.403595 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403515 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-slash\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.403595 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403531 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-etc-selinux\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.403595 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403532 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-run-ovn-kubernetes\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.403595 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403553 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b083c3f1-9c74-4313-a939-ff8aa5769c8d-ovnkube-config\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.403595 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403576 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-registration-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.403595 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403597 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-systemd\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.403595 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403600 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b91ad19-9236-4395-9e54-09470929bf35-cnibin\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.404282 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403618 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-run\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.404282 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403640 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-cnibin\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.404282 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403664 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-etc-kubernetes\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.404282 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403706 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b083c3f1-9c74-4313-a939-ff8aa5769c8d-ovnkube-script-lib\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.404282 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403726 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-sys\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.404282 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403746 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-daemon-config\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.404282 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403771 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-run-multus-certs\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.404282 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403787 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ed8a22-43d0-4ca3-9f7b-900231ad3685-tmp-dir\") pod \"node-resolver-ht26h\" (UID: \"a1ed8a22-43d0-4ca3-9f7b-900231ad3685\") " pod="openshift-dns/node-resolver-ht26h" Nov 02 01:24:02.404282 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403840 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-systemd-units\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.404282 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403884 2567 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 01:24:02.404282 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.403792 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg9vq\" (UniqueName: \"kubernetes.io/projected/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-kube-api-access-jg9vq\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.404282 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404137 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b91ad19-9236-4395-9e54-09470929bf35-cni-binary-copy\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.404282 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404146 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-kubelet\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.404282 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404180 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-etc-openvswitch\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.404282 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404208 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-socket-dir-parent\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.404282 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404220 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/673cb927-a5fc-4237-9b4c-fb196801d70c-iptables-alerter-script\") pod \"iptables-alerter-zzm69\" (UID: \"673cb927-a5fc-4237-9b4c-fb196801d70c\") " pod="openshift-network-operator/iptables-alerter-zzm69" Nov 02 01:24:02.404282 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404236 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-run-netns\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.404282 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404279 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-registration-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.405126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404220 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-etc-kubernetes\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.405126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404289 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-run-netns\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.405126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404312 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-run\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.405126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404353 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-cnibin\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.405126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404360 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-sys\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.405126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404368 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-etc-selinux\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.405126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404369 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b083c3f1-9c74-4313-a939-ff8aa5769c8d-ovnkube-config\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.405126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404403 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-systemd\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.405126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404412 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-tuned\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.405126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404426 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-run-multus-certs\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.405126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404434 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs\") pod \"network-metrics-daemon-flwvx\" (UID: \"d628518e-2ab7-4acb-acf7-6fc96f359a64\") " pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:02.405126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404442 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-etc-openvswitch\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.405126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404420 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-socket-dir-parent\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.405126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404483 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dzj7\" (UniqueName: \"kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7\") pod \"network-check-target-kgbsx\" (UID: \"85784876-9397-4bec-8ade-878a15c09f40\") " pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:02.405126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404488 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-kubelet\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.405126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404502 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79f97db3-814b-4849-ab8d-305dbcd3449c-kubelet-config\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:02.405126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404519 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b91ad19-9236-4395-9e54-09470929bf35-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.406003 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404535 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b91ad19-9236-4395-9e54-09470929bf35-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.406003 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404551 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-run-ovn\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.406003 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404570 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-cni-bin\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.406003 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404584 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrnrh\" (UniqueName: \"kubernetes.io/projected/673cb927-a5fc-4237-9b4c-fb196801d70c-kube-api-access-xrnrh\") pod \"iptables-alerter-zzm69\" (UID: \"673cb927-a5fc-4237-9b4c-fb196801d70c\") " pod="openshift-network-operator/iptables-alerter-zzm69" Nov 02 01:24:02.406003 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404597 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-conf-dir\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.406003 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404616 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crplh\" (UniqueName: \"kubernetes.io/projected/b083c3f1-9c74-4313-a939-ff8aa5769c8d-kube-api-access-crplh\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.406003 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404640 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4wlx\" (UniqueName: \"kubernetes.io/projected/a1ed8a22-43d0-4ca3-9f7b-900231ad3685-kube-api-access-b4wlx\") pod \"node-resolver-ht26h\" (UID: \"a1ed8a22-43d0-4ca3-9f7b-900231ad3685\") " pod="openshift-dns/node-resolver-ht26h" Nov 02 01:24:02.406003 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404663 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-sysconfig\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.406003 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404696 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zjgc\" (UniqueName: \"kubernetes.io/projected/e534da40-0733-4670-ba7a-839e0e4b1b4f-kube-api-access-5zjgc\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.406003 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404712 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh2r4\" (UniqueName: \"kubernetes.io/projected/d628518e-2ab7-4acb-acf7-6fc96f359a64-kube-api-access-rh2r4\") pod \"network-metrics-daemon-flwvx\" (UID: \"d628518e-2ab7-4acb-acf7-6fc96f359a64\") " pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:02.406003 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404717 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b083c3f1-9c74-4313-a939-ff8aa5769c8d-ovnkube-script-lib\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.406003 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404726 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79f97db3-814b-4849-ab8d-305dbcd3449c-dbus\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:02.406003 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404746 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b91ad19-9236-4395-9e54-09470929bf35-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.406003 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404774 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-var-lib-cni-bin\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.406003 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404789 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-sysconfig\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.406003 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404811 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-node-log\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.406003 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404822 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-var-lib-cni-bin\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.406654 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404847 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79f97db3-814b-4849-ab8d-305dbcd3449c-kubelet-config\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:02.406654 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404893 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-conf-dir\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.406654 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404951 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-run-ovn\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.406654 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404967 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-node-log\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.406654 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404968 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-daemon-config\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.406654 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.404852 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bb2a916a-fc15-4bdc-b28a-c2473a543ce5-konnectivity-ca\") pod \"konnectivity-agent-7mk45\" (UID: \"bb2a916a-fc15-4bdc-b28a-c2473a543ce5\") " pod="kube-system/konnectivity-agent-7mk45" Nov 02 01:24:02.406654 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405024 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-cni-bin\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.406654 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405032 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:02.406654 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405057 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-var-lib-kubelet\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.406654 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405082 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-run-systemd\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.406654 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405105 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-cni-netd\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.406654 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405118 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79f97db3-814b-4849-ab8d-305dbcd3449c-dbus\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:02.406654 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405130 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-724nj\" (UniqueName: \"kubernetes.io/projected/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-kube-api-access-724nj\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.406654 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405155 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-sysctl-d\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.406654 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405168 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-var-lib-kubelet\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.406654 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405180 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqppx\" (UniqueName: \"kubernetes.io/projected/9b91ad19-9236-4395-9e54-09470929bf35-kube-api-access-lqppx\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.406654 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405207 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-cni-binary-copy\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.407204 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405231 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-var-lib-cni-multus\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.407204 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:02.405243 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 01:24:02.407204 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405253 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b083c3f1-9c74-4313-a939-ff8aa5769c8d-ovn-node-metrics-cert\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.407204 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405274 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-sys-fs\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.407204 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405303 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-var-lib-kubelet\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.407204 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405327 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b91ad19-9236-4395-9e54-09470929bf35-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.407204 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:02.405342 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret podName:79f97db3-814b-4849-ab8d-305dbcd3449c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:02.905322208 +0000 UTC m=+3.067731129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret") pod "global-pull-secret-syncer-7szx6" (UID: "79f97db3-814b-4849-ab8d-305dbcd3449c") : object "kube-system"/"original-pull-secret" not registered Nov 02 01:24:02.407204 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405328 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-run-netns\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.407204 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405371 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-run-netns\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.407204 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405375 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-hostroot\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.407204 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405384 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bb2a916a-fc15-4bdc-b28a-c2473a543ce5-konnectivity-ca\") pod \"konnectivity-agent-7mk45\" (UID: \"bb2a916a-fc15-4bdc-b28a-c2473a543ce5\") " pod="kube-system/konnectivity-agent-7mk45" Nov 02 01:24:02.407204 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405410 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-hostroot\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.407204 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405416 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-cni-netd\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.407204 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405387 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-run-systemd\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.407204 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405442 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-run-openvswitch\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.407204 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405447 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b91ad19-9236-4395-9e54-09470929bf35-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.407204 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405469 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.407736 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405480 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-var-lib-kubelet\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.407736 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405494 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.407736 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405519 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1ed8a22-43d0-4ca3-9f7b-900231ad3685-hosts-file\") pod \"node-resolver-ht26h\" (UID: \"a1ed8a22-43d0-4ca3-9f7b-900231ad3685\") " pod="openshift-dns/node-resolver-ht26h" Nov 02 01:24:02.407736 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405544 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-modprobe-d\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.407736 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405567 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b91ad19-9236-4395-9e54-09470929bf35-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.407736 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405569 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-kubernetes\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.407736 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405582 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-sysctl-d\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.407736 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405609 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-log-socket\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.407736 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405618 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-kubernetes\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.407736 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405630 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1ed8a22-43d0-4ca3-9f7b-900231ad3685-hosts-file\") pod \"node-resolver-ht26h\" (UID: \"a1ed8a22-43d0-4ca3-9f7b-900231ad3685\") " pod="openshift-dns/node-resolver-ht26h" Nov 02 01:24:02.407736 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405634 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e534da40-0733-4670-ba7a-839e0e4b1b4f-tmp\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.407736 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405491 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-sys-fs\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.407736 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405659 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-run-openvswitch\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.407736 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405660 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.407736 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405682 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-run-k8s-cni-cncf-io\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.407736 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405619 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.407736 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405716 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a128d9e6-7be3-4c73-88f3-ba6b9ea59971-host\") pod \"node-ca-rrf9f\" (UID: \"a128d9e6-7be3-4c73-88f3-ba6b9ea59971\") " pod="openshift-image-registry/node-ca-rrf9f" Nov 02 01:24:02.408265 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405731 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-run-k8s-cni-cncf-io\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.408265 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405765 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-modprobe-d\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.408265 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405772 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-var-lib-cni-multus\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.408265 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405794 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a128d9e6-7be3-4c73-88f3-ba6b9ea59971-serviceca\") pod \"node-ca-rrf9f\" (UID: \"a128d9e6-7be3-4c73-88f3-ba6b9ea59971\") " pod="openshift-image-registry/node-ca-rrf9f" Nov 02 01:24:02.408265 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405809 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a128d9e6-7be3-4c73-88f3-ba6b9ea59971-host\") pod \"node-ca-rrf9f\" (UID: \"a128d9e6-7be3-4c73-88f3-ba6b9ea59971\") " pod="openshift-image-registry/node-ca-rrf9f" Nov 02 01:24:02.408265 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405821 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-log-socket\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.408265 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405848 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j98n8\" (UniqueName: \"kubernetes.io/projected/a128d9e6-7be3-4c73-88f3-ba6b9ea59971-kube-api-access-j98n8\") pod \"node-ca-rrf9f\" (UID: \"a128d9e6-7be3-4c73-88f3-ba6b9ea59971\") " pod="openshift-image-registry/node-ca-rrf9f" Nov 02 01:24:02.408265 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:02.405909 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:24:02.408265 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405924 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b083c3f1-9c74-4313-a939-ff8aa5769c8d-env-overrides\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.408265 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405944 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-cni-binary-copy\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.408265 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:02.405967 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs podName:d628518e-2ab7-4acb-acf7-6fc96f359a64 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:02.905951483 +0000 UTC m=+3.068360407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs") pod "network-metrics-daemon-flwvx" (UID: "d628518e-2ab7-4acb-acf7-6fc96f359a64") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:24:02.408265 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.405985 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-device-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.408265 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.406097 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-sysctl-conf\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.408265 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.406124 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-host\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.408265 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.406163 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/673cb927-a5fc-4237-9b4c-fb196801d70c-host-slash\") pod \"iptables-alerter-zzm69\" (UID: \"673cb927-a5fc-4237-9b4c-fb196801d70c\") " pod="openshift-network-operator/iptables-alerter-zzm69" Nov 02 01:24:02.408265 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.406188 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-system-cni-dir\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.408265 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.406197 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-device-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.408743 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.406215 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-cni-dir\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.408743 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.406233 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a128d9e6-7be3-4c73-88f3-ba6b9ea59971-serviceca\") pod \"node-ca-rrf9f\" (UID: \"a128d9e6-7be3-4c73-88f3-ba6b9ea59971\") " pod="openshift-image-registry/node-ca-rrf9f" Nov 02 01:24:02.408743 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.406287 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-host\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.408743 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.406303 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-system-cni-dir\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.408743 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.406321 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-cni-dir\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.408743 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.406327 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/673cb927-a5fc-4237-9b4c-fb196801d70c-host-slash\") pod \"iptables-alerter-zzm69\" (UID: \"673cb927-a5fc-4237-9b4c-fb196801d70c\") " pod="openshift-network-operator/iptables-alerter-zzm69" Nov 02 01:24:02.408743 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.406324 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b083c3f1-9c74-4313-a939-ff8aa5769c8d-env-overrides\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.408743 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.406446 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-sysctl-conf\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.408743 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.407847 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bb2a916a-fc15-4bdc-b28a-c2473a543ce5-agent-certs\") pod \"konnectivity-agent-7mk45\" (UID: \"bb2a916a-fc15-4bdc-b28a-c2473a543ce5\") " pod="kube-system/konnectivity-agent-7mk45" Nov 02 01:24:02.408743 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.408065 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-tuned\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.408743 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.408098 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e534da40-0733-4670-ba7a-839e0e4b1b4f-tmp\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.408743 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.408269 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b083c3f1-9c74-4313-a939-ff8aa5769c8d-ovn-node-metrics-cert\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.462299 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:02.462218 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 01:24:02.462299 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:02.462244 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 01:24:02.462299 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:02.462265 2567 projected.go:196] Error preparing data for projected volume kube-api-access-4dzj7 for pod openshift-network-diagnostics/network-check-target-kgbsx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:24:02.462492 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:02.462340 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7 podName:85784876-9397-4bec-8ade-878a15c09f40 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:02.962318696 +0000 UTC m=+3.124727616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4dzj7" (UniqueName: "kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7") pod "network-check-target-kgbsx" (UID: "85784876-9397-4bec-8ade-878a15c09f40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:24:02.464803 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.464779 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrnrh\" (UniqueName: \"kubernetes.io/projected/673cb927-a5fc-4237-9b4c-fb196801d70c-kube-api-access-xrnrh\") pod \"iptables-alerter-zzm69\" (UID: \"673cb927-a5fc-4237-9b4c-fb196801d70c\") " pod="openshift-network-operator/iptables-alerter-zzm69" Nov 02 01:24:02.464990 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.464821 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zjgc\" (UniqueName: \"kubernetes.io/projected/e534da40-0733-4670-ba7a-839e0e4b1b4f-kube-api-access-5zjgc\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.464990 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.464968 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg9vq\" (UniqueName: \"kubernetes.io/projected/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-kube-api-access-jg9vq\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.465497 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.465470 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4wlx\" (UniqueName: \"kubernetes.io/projected/a1ed8a22-43d0-4ca3-9f7b-900231ad3685-kube-api-access-b4wlx\") pod \"node-resolver-ht26h\" (UID: \"a1ed8a22-43d0-4ca3-9f7b-900231ad3685\") " pod="openshift-dns/node-resolver-ht26h" Nov 02 01:24:02.466270 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.466237 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-724nj\" (UniqueName: \"kubernetes.io/projected/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-kube-api-access-724nj\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.466518 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.466500 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crplh\" (UniqueName: \"kubernetes.io/projected/b083c3f1-9c74-4313-a939-ff8aa5769c8d-kube-api-access-crplh\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.466588 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.466533 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqppx\" (UniqueName: \"kubernetes.io/projected/9b91ad19-9236-4395-9e54-09470929bf35-kube-api-access-lqppx\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.466662 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.466647 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh2r4\" (UniqueName: \"kubernetes.io/projected/d628518e-2ab7-4acb-acf7-6fc96f359a64-kube-api-access-rh2r4\") pod \"network-metrics-daemon-flwvx\" (UID: \"d628518e-2ab7-4acb-acf7-6fc96f359a64\") " pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:02.467141 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.467120 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j98n8\" (UniqueName: \"kubernetes.io/projected/a128d9e6-7be3-4c73-88f3-ba6b9ea59971-kube-api-access-j98n8\") pod \"node-ca-rrf9f\" (UID: \"a128d9e6-7be3-4c73-88f3-ba6b9ea59971\") " pod="openshift-image-registry/node-ca-rrf9f" Nov 02 01:24:02.594701 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.594661 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:02.595750 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.595729 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wzzsb" Nov 02 01:24:02.603144 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.603127 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:02.606431 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:02.606403 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b91ad19_9236_4395_9e54_09470929bf35.slice/crio-fb877fe27b5a235f783aeb6c121b0d7df45af00d84d72cacb0272dec49d6fd58 WatchSource:0}: Error finding container fb877fe27b5a235f783aeb6c121b0d7df45af00d84d72cacb0272dec49d6fd58: Status 404 returned error can't find the container with id fb877fe27b5a235f783aeb6c121b0d7df45af00d84d72cacb0272dec49d6fd58 Nov 02 01:24:02.607774 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.607754 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zzm69" Nov 02 01:24:02.613619 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.613598 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rrf9f" Nov 02 01:24:02.613872 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:02.613851 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode534da40_0733_4670_ba7a_839e0e4b1b4f.slice/crio-6774db5530a54d2006e805f898ce88c9809659c85a5cf6928535d127a7c89427 WatchSource:0}: Error finding container 6774db5530a54d2006e805f898ce88c9809659c85a5cf6928535d127a7c89427: Status 404 returned error can't find the container with id 6774db5530a54d2006e805f898ce88c9809659c85a5cf6928535d127a7c89427 Nov 02 01:24:02.616491 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:02.616471 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod673cb927_a5fc_4237_9b4c_fb196801d70c.slice/crio-f28823619809a7408c348ae1039f3bde4f82f41d60899137778532d11d51718c WatchSource:0}: Error finding container f28823619809a7408c348ae1039f3bde4f82f41d60899137778532d11d51718c: Status 404 returned error can't find the container with id f28823619809a7408c348ae1039f3bde4f82f41d60899137778532d11d51718c Nov 02 01:24:02.620448 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:02.620429 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda128d9e6_7be3_4c73_88f3_ba6b9ea59971.slice/crio-5b45cf67f3d40c4644ea1e2a59eeb281e816e65de1f906142311859c233cb806 WatchSource:0}: Error finding container 5b45cf67f3d40c4644ea1e2a59eeb281e816e65de1f906142311859c233cb806: Status 404 returned error can't find the container with id 5b45cf67f3d40c4644ea1e2a59eeb281e816e65de1f906142311859c233cb806 Nov 02 01:24:02.625381 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.625366 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:02.628237 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.628224 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:02.630617 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.630593 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7mk45" Nov 02 01:24:02.632860 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:02.632836 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb083c3f1_9c74_4313_a939_ff8aa5769c8d.slice/crio-cae0ce038d06591c037675c225f487723b7f02925e0eddf2839d06909619ccbb WatchSource:0}: Error finding container cae0ce038d06591c037675c225f487723b7f02925e0eddf2839d06909619ccbb: Status 404 returned error can't find the container with id cae0ce038d06591c037675c225f487723b7f02925e0eddf2839d06909619ccbb Nov 02 01:24:02.635051 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.635031 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ht26h" Nov 02 01:24:02.637272 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:02.637252 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25437e9b_ca22_4103_ac9e_c5823f2bbbaf.slice/crio-624e8843c916c2fcc231815c561ff5315e6251132d93db11dbffb2278bb17833 WatchSource:0}: Error finding container 624e8843c916c2fcc231815c561ff5315e6251132d93db11dbffb2278bb17833: Status 404 returned error can't find the container with id 624e8843c916c2fcc231815c561ff5315e6251132d93db11dbffb2278bb17833 Nov 02 01:24:02.637873 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:02.637854 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2a916a_fc15_4bdc_b28a_c2473a543ce5.slice/crio-dbbaa485276e9e80414df1c76559d8406a6ebf38307eccac18eda98c904bffde WatchSource:0}: Error finding container dbbaa485276e9e80414df1c76559d8406a6ebf38307eccac18eda98c904bffde: Status 404 returned error can't find the container with id dbbaa485276e9e80414df1c76559d8406a6ebf38307eccac18eda98c904bffde Nov 02 01:24:02.643843 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:02.643821 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ed8a22_43d0_4ca3_9f7b_900231ad3685.slice/crio-671ea2856c9b53a7db137f865f26d6585cd38fb6098f982fdcc8526857b37f9f WatchSource:0}: Error finding container 671ea2856c9b53a7db137f865f26d6585cd38fb6098f982fdcc8526857b37f9f: Status 404 returned error can't find the container with id 671ea2856c9b53a7db137f865f26d6585cd38fb6098f982fdcc8526857b37f9f Nov 02 01:24:02.909590 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.909511 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs\") pod \"network-metrics-daemon-flwvx\" (UID: \"d628518e-2ab7-4acb-acf7-6fc96f359a64\") " pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:02.909590 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:02.909572 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:02.909832 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:02.909683 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:24:02.909832 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:02.909745 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs podName:d628518e-2ab7-4acb-acf7-6fc96f359a64 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:03.909726944 +0000 UTC m=+4.072135869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs") pod "network-metrics-daemon-flwvx" (UID: "d628518e-2ab7-4acb-acf7-6fc96f359a64") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:24:02.909832 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:02.909687 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 01:24:02.909832 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:02.909790 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret podName:79f97db3-814b-4849-ab8d-305dbcd3449c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:03.909780014 +0000 UTC m=+4.072188939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret") pod "global-pull-secret-syncer-7szx6" (UID: "79f97db3-814b-4849-ab8d-305dbcd3449c") : object "kube-system"/"original-pull-secret" not registered Nov 02 01:24:03.010864 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:03.010828 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dzj7\" (UniqueName: \"kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7\") pod \"network-check-target-kgbsx\" (UID: \"85784876-9397-4bec-8ade-878a15c09f40\") " pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:03.011037 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:03.011017 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 01:24:03.011037 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:03.011035 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 01:24:03.011119 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:03.011049 2567 projected.go:196] Error preparing data for projected volume kube-api-access-4dzj7 for pod openshift-network-diagnostics/network-check-target-kgbsx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:24:03.011119 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:03.011105 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7 podName:85784876-9397-4bec-8ade-878a15c09f40 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:04.011086 +0000 UTC m=+4.173494927 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dzj7" (UniqueName: "kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7") pod "network-check-target-kgbsx" (UID: "85784876-9397-4bec-8ade-878a15c09f40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:24:03.267857 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:03.267787 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ht26h" event={"ID":"a1ed8a22-43d0-4ca3-9f7b-900231ad3685","Type":"ContainerStarted","Data":"671ea2856c9b53a7db137f865f26d6585cd38fb6098f982fdcc8526857b37f9f"} Nov 02 01:24:03.269736 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:03.269699 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7mk45" event={"ID":"bb2a916a-fc15-4bdc-b28a-c2473a543ce5","Type":"ContainerStarted","Data":"dbbaa485276e9e80414df1c76559d8406a6ebf38307eccac18eda98c904bffde"} Nov 02 01:24:03.270965 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:03.270938 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" event={"ID":"25437e9b-ca22-4103-ac9e-c5823f2bbbaf","Type":"ContainerStarted","Data":"624e8843c916c2fcc231815c561ff5315e6251132d93db11dbffb2278bb17833"} Nov 02 01:24:03.272183 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:03.272157 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rrf9f" event={"ID":"a128d9e6-7be3-4c73-88f3-ba6b9ea59971","Type":"ContainerStarted","Data":"5b45cf67f3d40c4644ea1e2a59eeb281e816e65de1f906142311859c233cb806"} Nov 02 01:24:03.273386 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:03.273366 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zzm69" event={"ID":"673cb927-a5fc-4237-9b4c-fb196801d70c","Type":"ContainerStarted","Data":"f28823619809a7408c348ae1039f3bde4f82f41d60899137778532d11d51718c"} Nov 02 01:24:03.274469 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:03.274448 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wzzsb" event={"ID":"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99","Type":"ContainerStarted","Data":"2d604ba2867e44ccc9b62c8fe569b770ac1a144bca0d2ddfedade3e526a2ff0b"} Nov 02 01:24:03.275862 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:03.275840 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" event={"ID":"b083c3f1-9c74-4313-a939-ff8aa5769c8d","Type":"ContainerStarted","Data":"cae0ce038d06591c037675c225f487723b7f02925e0eddf2839d06909619ccbb"} Nov 02 01:24:03.277094 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:03.277074 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" event={"ID":"e534da40-0733-4670-ba7a-839e0e4b1b4f","Type":"ContainerStarted","Data":"6774db5530a54d2006e805f898ce88c9809659c85a5cf6928535d127a7c89427"} Nov 02 01:24:03.278271 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:03.278244 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mgjgw" event={"ID":"9b91ad19-9236-4395-9e54-09470929bf35","Type":"ContainerStarted","Data":"fb877fe27b5a235f783aeb6c121b0d7df45af00d84d72cacb0272dec49d6fd58"} Nov 02 01:24:03.382987 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:03.382927 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:19:02 +0000 UTC" deadline="2028-06-30 23:13:40.601145431 +0000 UTC" Nov 02 01:24:03.382987 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:03.382979 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14565h49m37.218170606s" Nov 02 01:24:03.920236 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:03.920203 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs\") pod \"network-metrics-daemon-flwvx\" (UID: \"d628518e-2ab7-4acb-acf7-6fc96f359a64\") " pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:03.920410 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:03.920264 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:03.920410 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:03.920395 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 01:24:03.920538 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:03.920452 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret podName:79f97db3-814b-4849-ab8d-305dbcd3449c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:05.920435623 +0000 UTC m=+6.082844556 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret") pod "global-pull-secret-syncer-7szx6" (UID: "79f97db3-814b-4849-ab8d-305dbcd3449c") : object "kube-system"/"original-pull-secret" not registered Nov 02 01:24:03.920538 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:03.920515 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:24:03.920725 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:03.920551 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs podName:d628518e-2ab7-4acb-acf7-6fc96f359a64 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:05.920538799 +0000 UTC m=+6.082947733 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs") pod "network-metrics-daemon-flwvx" (UID: "d628518e-2ab7-4acb-acf7-6fc96f359a64") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:24:04.021485 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:04.021450 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dzj7\" (UniqueName: \"kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7\") pod \"network-check-target-kgbsx\" (UID: \"85784876-9397-4bec-8ade-878a15c09f40\") " pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:04.021657 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:04.021640 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 01:24:04.021761 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:04.021683 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 01:24:04.021761 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:04.021698 2567 projected.go:196] Error preparing data for projected volume kube-api-access-4dzj7 for pod openshift-network-diagnostics/network-check-target-kgbsx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:24:04.021761 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:04.021757 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7 podName:85784876-9397-4bec-8ade-878a15c09f40 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:06.021740268 +0000 UTC m=+6.184149205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dzj7" (UniqueName: "kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7") pod "network-check-target-kgbsx" (UID: "85784876-9397-4bec-8ade-878a15c09f40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:24:04.259769 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:04.259222 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:04.259769 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:04.259342 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgbsx" podUID="85784876-9397-4bec-8ade-878a15c09f40" Nov 02 01:24:04.259769 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:04.259439 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:04.259769 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:04.259518 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7szx6" podUID="79f97db3-814b-4849-ab8d-305dbcd3449c" Nov 02 01:24:04.259769 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:04.259582 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:04.259769 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:04.259659 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flwvx" podUID="d628518e-2ab7-4acb-acf7-6fc96f359a64" Nov 02 01:24:04.383788 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:04.383747 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:19:02 +0000 UTC" deadline="2028-06-02 22:00:35.952609543 +0000 UTC" Nov 02 01:24:04.384139 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:04.383790 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13892h36m31.568824096s" Nov 02 01:24:05.296364 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:05.296132 2567 generic.go:356] "Generic (PLEG): container finished" podID="1c52cca1e001b790fd8c346cdd80a9e7" containerID="604c7bf3cede74e00e8560f530e4c9bdb32b7262cd71a8a5a47cee2083c835f0" exitCode=0 Nov 02 01:24:05.296518 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:05.296456 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" event={"ID":"1c52cca1e001b790fd8c346cdd80a9e7","Type":"ContainerDied","Data":"604c7bf3cede74e00e8560f530e4c9bdb32b7262cd71a8a5a47cee2083c835f0"} Nov 02 01:24:05.326648 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:05.326608 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal" event={"ID":"817dd2f8fadd21a6e3346c963f0534ec","Type":"ContainerStarted","Data":"729b6926b84ebd4b24e5750dfca74a28f4f321eb1f6a82601d39b4fa8be6a036"} Nov 02 01:24:05.941094 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:05.941018 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs\") pod \"network-metrics-daemon-flwvx\" (UID: \"d628518e-2ab7-4acb-acf7-6fc96f359a64\") " pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:05.941510 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:05.941097 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:05.941510 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:05.941189 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:24:05.941510 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:05.941231 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 01:24:05.941510 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:05.941271 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs podName:d628518e-2ab7-4acb-acf7-6fc96f359a64 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:09.941249689 +0000 UTC m=+10.103658611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs") pod "network-metrics-daemon-flwvx" (UID: "d628518e-2ab7-4acb-acf7-6fc96f359a64") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:24:05.941510 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:05.941291 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret podName:79f97db3-814b-4849-ab8d-305dbcd3449c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:09.94128144 +0000 UTC m=+10.103690366 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret") pod "global-pull-secret-syncer-7szx6" (UID: "79f97db3-814b-4849-ab8d-305dbcd3449c") : object "kube-system"/"original-pull-secret" not registered Nov 02 01:24:06.042601 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:06.042562 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dzj7\" (UniqueName: \"kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7\") pod \"network-check-target-kgbsx\" (UID: \"85784876-9397-4bec-8ade-878a15c09f40\") " pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:06.042787 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:06.042766 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 01:24:06.042787 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:06.042786 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 01:24:06.042908 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:06.042801 2567 projected.go:196] Error preparing data for projected volume kube-api-access-4dzj7 for pod openshift-network-diagnostics/network-check-target-kgbsx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:24:06.042908 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:06.042856 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7 podName:85784876-9397-4bec-8ade-878a15c09f40 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:10.042838127 +0000 UTC m=+10.205247061 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dzj7" (UniqueName: "kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7") pod "network-check-target-kgbsx" (UID: "85784876-9397-4bec-8ade-878a15c09f40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:24:06.261015 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:06.260983 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:06.261178 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:06.261120 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flwvx" podUID="d628518e-2ab7-4acb-acf7-6fc96f359a64" Nov 02 01:24:06.261259 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:06.261186 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:06.261259 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:06.261233 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgbsx" podUID="85784876-9397-4bec-8ade-878a15c09f40" Nov 02 01:24:06.261367 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:06.261270 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:06.261367 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:06.261310 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7szx6" podUID="79f97db3-814b-4849-ab8d-305dbcd3449c" Nov 02 01:24:06.338403 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:06.338325 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" event={"ID":"1c52cca1e001b790fd8c346cdd80a9e7","Type":"ContainerStarted","Data":"17b51a45675b4dcb0bae224833d657cc961be6839ead272ca5fb690d2d35d12b"} Nov 02 01:24:06.453339 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:06.453219 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal" podStartSLOduration=5.453199501 podStartE2EDuration="5.453199501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:05.525812737 +0000 UTC m=+5.688221681" watchObservedRunningTime="2026-11-02 01:24:06.453199501 +0000 UTC m=+6.615608445" Nov 02 01:24:07.346090 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:07.346053 2567 generic.go:356] "Generic (PLEG): container finished" podID="9b91ad19-9236-4395-9e54-09470929bf35" containerID="83ef0f032e62a4c342d3b8a2fa490baa32f5b8ed7e8b550b6acb686623c8836a" exitCode=0 Nov 02 01:24:07.347105 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:07.347072 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mgjgw" event={"ID":"9b91ad19-9236-4395-9e54-09470929bf35","Type":"ContainerDied","Data":"83ef0f032e62a4c342d3b8a2fa490baa32f5b8ed7e8b550b6acb686623c8836a"} Nov 02 01:24:07.468433 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:07.467344 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" podStartSLOduration=6.467325342 podStartE2EDuration="6.467325342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:06.454180503 +0000 UTC m=+6.616589445" watchObservedRunningTime="2026-11-02 01:24:07.467325342 +0000 UTC m=+7.629734287" Nov 02 01:24:08.259811 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:08.259789 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:08.259942 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:08.259919 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flwvx" podUID="d628518e-2ab7-4acb-acf7-6fc96f359a64" Nov 02 01:24:08.260305 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:08.260287 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:08.260403 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:08.260384 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgbsx" podUID="85784876-9397-4bec-8ade-878a15c09f40" Nov 02 01:24:08.260477 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:08.260441 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:08.260535 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:08.260513 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7szx6" podUID="79f97db3-814b-4849-ab8d-305dbcd3449c" Nov 02 01:24:08.353449 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:08.353408 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7mk45" event={"ID":"bb2a916a-fc15-4bdc-b28a-c2473a543ce5","Type":"ContainerStarted","Data":"31837c48249c2783315f676779f6dc7ba75aa724dccc16572e31e6e9d14c73ca"} Nov 02 01:24:08.358829 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:08.358799 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rrf9f" event={"ID":"a128d9e6-7be3-4c73-88f3-ba6b9ea59971","Type":"ContainerStarted","Data":"3fe30dd463550262ae99deb94a4beb77ea6b01d08b110b0a58362f7c8e873ab3"} Nov 02 01:24:08.582271 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:08.582164 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7mk45" podStartSLOduration=3.005446084 podStartE2EDuration="7.582142863s" totalImagesPullingTime="4.576696779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:08.468344041 +0000 UTC m=+8.630752995" watchObservedRunningTime="2026-11-02 01:24:08.582142863 +0000 UTC m=+8.744551806" Nov 02 01:24:09.364102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:09.364071 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" event={"ID":"25437e9b-ca22-4103-ac9e-c5823f2bbbaf","Type":"ContainerStarted","Data":"528f24b2412bc60a5eab712b21917a04cb335f38c4381a8d2015a3343e83d09c"} Nov 02 01:24:09.976917 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:09.976835 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs\") pod \"network-metrics-daemon-flwvx\" (UID: \"d628518e-2ab7-4acb-acf7-6fc96f359a64\") " pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:09.977117 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:09.976932 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:09.977117 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:09.977057 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 01:24:09.977117 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:09.977116 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret podName:79f97db3-814b-4849-ab8d-305dbcd3449c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:17.977099811 +0000 UTC m=+18.139508745 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret") pod "global-pull-secret-syncer-7szx6" (UID: "79f97db3-814b-4849-ab8d-305dbcd3449c") : object "kube-system"/"original-pull-secret" not registered Nov 02 01:24:09.977725 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:09.977705 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:24:09.977845 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:09.977789 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs podName:d628518e-2ab7-4acb-acf7-6fc96f359a64 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:17.977775614 +0000 UTC m=+18.140184535 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs") pod "network-metrics-daemon-flwvx" (UID: "d628518e-2ab7-4acb-acf7-6fc96f359a64") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:24:10.077977 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:10.077909 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dzj7\" (UniqueName: \"kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7\") pod \"network-check-target-kgbsx\" (UID: \"85784876-9397-4bec-8ade-878a15c09f40\") " pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:10.078147 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:10.078119 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 01:24:10.078147 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:10.078139 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 01:24:10.078147 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:10.078152 2567 projected.go:196] Error preparing data for projected volume kube-api-access-4dzj7 for pod openshift-network-diagnostics/network-check-target-kgbsx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:24:10.078316 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:10.078217 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7 podName:85784876-9397-4bec-8ade-878a15c09f40 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:18.078198087 +0000 UTC m=+18.240607013 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dzj7" (UniqueName: "kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7") pod "network-check-target-kgbsx" (UID: "85784876-9397-4bec-8ade-878a15c09f40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:24:10.261060 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:10.258993 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:10.261060 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:10.259129 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flwvx" podUID="d628518e-2ab7-4acb-acf7-6fc96f359a64" Nov 02 01:24:10.261060 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:10.259745 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:10.261060 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:10.259865 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgbsx" podUID="85784876-9397-4bec-8ade-878a15c09f40" Nov 02 01:24:10.261060 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:10.259933 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:10.261060 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:10.259980 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7szx6" podUID="79f97db3-814b-4849-ab8d-305dbcd3449c" Nov 02 01:24:11.015923 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:11.015724 2567 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 01:24:11.259894 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:11.259792 2567 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T01:24:11.015919751Z","UUID":"7a5dc029-960b-46a5-9cd0-8743f4fdeede","Handler":null,"Name":"","Endpoint":""} Nov 02 01:24:11.261486 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:11.261450 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 01:24:11.261486 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:11.261479 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 01:24:11.323685 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:11.323643 2567 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7mk45" Nov 02 01:24:11.324332 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:11.324313 2567 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7mk45" Nov 02 01:24:11.369597 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:11.369529 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ht26h" event={"ID":"a1ed8a22-43d0-4ca3-9f7b-900231ad3685","Type":"ContainerStarted","Data":"dec59348470a58e5c6e6bf431377ccd158d9308a01a657aff819a69ca2e52e9b"} Nov 02 01:24:11.372501 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:11.372471 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" event={"ID":"25437e9b-ca22-4103-ac9e-c5823f2bbbaf","Type":"ContainerStarted","Data":"577cbe6164059e0462cbd0f67babb11c1255aff8d5f9d0fd4dfc8153fe0e2049"} Nov 02 01:24:11.375250 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:11.375228 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wzzsb" event={"ID":"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99","Type":"ContainerStarted","Data":"becaf38f2512475763f15256644bd5f745f594506115eef19453e53ee4bbf772"} Nov 02 01:24:11.375349 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:11.375270 2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7mk45" Nov 02 01:24:11.375774 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:11.375754 2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7mk45" Nov 02 01:24:11.440860 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:11.440806 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rrf9f" podStartSLOduration=5.927228295 podStartE2EDuration="11.440789981s" totalImagesPullingTime="5.513561686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:08.583096435 +0000 UTC m=+8.745505378" watchObservedRunningTime="2026-11-02 01:24:11.440789981 +0000 UTC m=+11.603198924" Nov 02 01:24:11.558557 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:11.558495 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-wzzsb" podStartSLOduration=3.633336962 podStartE2EDuration="11.558475261s" totalImagesPullingTime="7.925138299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:11.557512737 +0000 UTC m=+11.719921680" watchObservedRunningTime="2026-11-02 01:24:11.558475261 +0000 UTC m=+11.720884204" Nov 02 01:24:11.787976 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:11.787919 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-ht26h" podStartSLOduration=2.587551236 podStartE2EDuration="10.787897502s" totalImagesPullingTime="8.200346266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:11.787554249 +0000 UTC m=+11.949963190" watchObservedRunningTime="2026-11-02 01:24:11.787897502 +0000 UTC m=+11.950306446" Nov 02 01:24:12.258790 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:12.258695 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:12.258790 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:12.258765 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:12.258790 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:12.258762 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:12.259437 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:12.258867 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgbsx" podUID="85784876-9397-4bec-8ade-878a15c09f40" Nov 02 01:24:12.259437 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:12.258979 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7szx6" podUID="79f97db3-814b-4849-ab8d-305dbcd3449c" Nov 02 01:24:12.259437 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:12.259049 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flwvx" podUID="d628518e-2ab7-4acb-acf7-6fc96f359a64" Nov 02 01:24:12.380425 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:12.380394 2567 generic.go:356] "Generic (PLEG): container finished" podID="9b91ad19-9236-4395-9e54-09470929bf35" containerID="f19f28adba58efff0ae92b9429561293b64faa1dcf6481af6591de81d51ce982" exitCode=0 Nov 02 01:24:12.380585 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:12.380496 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mgjgw" event={"ID":"9b91ad19-9236-4395-9e54-09470929bf35","Type":"ContainerDied","Data":"f19f28adba58efff0ae92b9429561293b64faa1dcf6481af6591de81d51ce982"} Nov 02 01:24:13.385601 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:13.385526 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" event={"ID":"25437e9b-ca22-4103-ac9e-c5823f2bbbaf","Type":"ContainerStarted","Data":"7ef67e42e4c5eee8be9a5613c6e65a93f9b4f277cb450eba06e0218d9eba6454"} Nov 02 01:24:13.387562 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:13.387527 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zzm69" event={"ID":"673cb927-a5fc-4237-9b4c-fb196801d70c","Type":"ContainerStarted","Data":"4590d397787d1d4e2f670fb17b84649dc3a511e9c893043f7474cee325952c84"} Nov 02 01:24:13.389134 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:13.389112 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" event={"ID":"e534da40-0733-4670-ba7a-839e0e4b1b4f","Type":"ContainerStarted","Data":"56302fed2e438ae9b20c460c428f091eb0f2f59cc6b157fc47d1cafe824aa06f"} Nov 02 01:24:13.504295 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:13.504246 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" podStartSLOduration=2.839954085 podStartE2EDuration="12.504232601s" totalImagesPullingTime="9.664278516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:13.503861062 +0000 UTC m=+13.666270004" watchObservedRunningTime="2026-11-02 01:24:13.504232601 +0000 UTC m=+13.666641543" Nov 02 01:24:13.663939 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:13.663878 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" podStartSLOduration=3.21695283 podStartE2EDuration="13.66385915s" totalImagesPullingTime="10.44690632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:13.659660583 +0000 UTC m=+13.822069527" watchObservedRunningTime="2026-11-02 01:24:13.66385915 +0000 UTC m=+13.826268093" Nov 02 01:24:13.774006 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:13.773835 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zzm69" podStartSLOduration=5.301687625 podStartE2EDuration="13.773821422s" totalImagesPullingTime="8.472133797s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:13.773534472 +0000 UTC m=+13.935943413" watchObservedRunningTime="2026-11-02 01:24:13.773821422 +0000 UTC m=+13.936230364" Nov 02 01:24:14.258732 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:14.258699 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:14.258732 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:14.258708 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:14.258956 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:14.258840 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:14.258956 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:14.258842 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgbsx" podUID="85784876-9397-4bec-8ade-878a15c09f40" Nov 02 01:24:14.258956 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:14.258943 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flwvx" podUID="d628518e-2ab7-4acb-acf7-6fc96f359a64" Nov 02 01:24:14.259108 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:14.259022 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7szx6" podUID="79f97db3-814b-4849-ab8d-305dbcd3449c" Nov 02 01:24:14.393122 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:14.393083 2567 generic.go:356] "Generic (PLEG): container finished" podID="9b91ad19-9236-4395-9e54-09470929bf35" containerID="ebf1caa1d56ea1dec1b3f2840f28ec0cc26651f2425fa52f99d4429b3d42625d" exitCode=0 Nov 02 01:24:14.393590 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:14.393195 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mgjgw" event={"ID":"9b91ad19-9236-4395-9e54-09470929bf35","Type":"ContainerDied","Data":"ebf1caa1d56ea1dec1b3f2840f28ec0cc26651f2425fa52f99d4429b3d42625d"} Nov 02 01:24:16.258613 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:16.258577 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:16.258613 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:16.258604 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:16.259065 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:16.258713 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:16.259065 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:16.258723 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgbsx" podUID="85784876-9397-4bec-8ade-878a15c09f40" Nov 02 01:24:16.259065 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:16.258847 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7szx6" podUID="79f97db3-814b-4849-ab8d-305dbcd3449c" Nov 02 01:24:16.259065 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:16.258947 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flwvx" podUID="d628518e-2ab7-4acb-acf7-6fc96f359a64" Nov 02 01:24:16.413467 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:16.413420 2567 generic.go:356] "Generic (PLEG): container finished" podID="9b91ad19-9236-4395-9e54-09470929bf35" containerID="52cfaac52b7d581bcf70568552e2938c6770de749284ffc6344f31fd3cbe0b29" exitCode=0 Nov 02 01:24:16.413613 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:16.413482 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mgjgw" event={"ID":"9b91ad19-9236-4395-9e54-09470929bf35","Type":"ContainerDied","Data":"52cfaac52b7d581bcf70568552e2938c6770de749284ffc6344f31fd3cbe0b29"} Nov 02 01:24:18.039471 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:18.039432 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs\") pod \"network-metrics-daemon-flwvx\" (UID: \"d628518e-2ab7-4acb-acf7-6fc96f359a64\") " pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:18.039912 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:18.039492 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:18.039912 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:18.039594 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 01:24:18.039912 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:18.039596 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:24:18.039912 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:18.039654 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret podName:79f97db3-814b-4849-ab8d-305dbcd3449c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:34.039636908 +0000 UTC m=+34.202045832 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret") pod "global-pull-secret-syncer-7szx6" (UID: "79f97db3-814b-4849-ab8d-305dbcd3449c") : object "kube-system"/"original-pull-secret" not registered Nov 02 01:24:18.039912 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:18.039686 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs podName:d628518e-2ab7-4acb-acf7-6fc96f359a64 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:34.039663734 +0000 UTC m=+34.202072658 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs") pod "network-metrics-daemon-flwvx" (UID: "d628518e-2ab7-4acb-acf7-6fc96f359a64") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 01:24:18.140199 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:18.140157 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dzj7\" (UniqueName: \"kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7\") pod \"network-check-target-kgbsx\" (UID: \"85784876-9397-4bec-8ade-878a15c09f40\") " pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:18.140348 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:18.140331 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 01:24:18.140395 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:18.140376 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 01:24:18.140440 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:18.140395 2567 projected.go:196] Error preparing data for projected volume kube-api-access-4dzj7 for pod openshift-network-diagnostics/network-check-target-kgbsx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:24:18.140487 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:18.140454 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7 podName:85784876-9397-4bec-8ade-878a15c09f40 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:34.140437766 +0000 UTC m=+34.302846697 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4dzj7" (UniqueName: "kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7") pod "network-check-target-kgbsx" (UID: "85784876-9397-4bec-8ade-878a15c09f40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 01:24:18.258924 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:18.258897 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:18.258924 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:18.258897 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:18.259144 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:18.258898 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:18.259144 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:18.259099 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgbsx" podUID="85784876-9397-4bec-8ade-878a15c09f40" Nov 02 01:24:18.259144 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:18.259081 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flwvx" podUID="d628518e-2ab7-4acb-acf7-6fc96f359a64" Nov 02 01:24:18.259144 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:18.259135 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7szx6" podUID="79f97db3-814b-4849-ab8d-305dbcd3449c" Nov 02 01:24:19.422732 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:19.422535 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" event={"ID":"b083c3f1-9c74-4313-a939-ff8aa5769c8d","Type":"ContainerStarted","Data":"408b3c7f7f2018fee8a58e01f1179913658313d14758d00b54ceb1acbebad657"} Nov 02 01:24:19.423356 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:19.422817 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" event={"ID":"b083c3f1-9c74-4313-a939-ff8aa5769c8d","Type":"ContainerStarted","Data":"727c0f1cd7ebbcb01173ab3171c48f0ec978435a285e8f6221410cda19195693"} Nov 02 01:24:19.423356 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:19.422866 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" event={"ID":"b083c3f1-9c74-4313-a939-ff8aa5769c8d","Type":"ContainerStarted","Data":"18480cdfb71dc8aaaea3b0abb8049291e1acfaf874850ad26589f39cdb3defcc"} Nov 02 01:24:19.423356 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:19.422909 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" event={"ID":"b083c3f1-9c74-4313-a939-ff8aa5769c8d","Type":"ContainerStarted","Data":"9a3e8cfadd695cd016b7ddd40d8a68dcec46e55be06c2a36a99173797e0a5649"} Nov 02 01:24:19.423356 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:19.422958 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" event={"ID":"b083c3f1-9c74-4313-a939-ff8aa5769c8d","Type":"ContainerStarted","Data":"c56eeb7e8827923f709966432b06184dc0b9170c7a58a1b91755a200beb4efb5"} Nov 02 01:24:19.423356 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:19.423008 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" event={"ID":"b083c3f1-9c74-4313-a939-ff8aa5769c8d","Type":"ContainerStarted","Data":"36feaba39d7150c3337730703525b4c1e74fa41fa0f352f9f59f5736a9205810"} Nov 02 01:24:19.427250 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:19.427224 2567 generic.go:356] "Generic (PLEG): container finished" podID="9b91ad19-9236-4395-9e54-09470929bf35" containerID="9921ab932f988037f47c0de6df7bbcb96fb3c29dbbd89ecc8ad4da7051f22bfc" exitCode=0 Nov 02 01:24:19.427364 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:19.427263 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mgjgw" event={"ID":"9b91ad19-9236-4395-9e54-09470929bf35","Type":"ContainerDied","Data":"9921ab932f988037f47c0de6df7bbcb96fb3c29dbbd89ecc8ad4da7051f22bfc"} Nov 02 01:24:20.259437 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:20.259406 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:20.259606 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:20.259488 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flwvx" podUID="d628518e-2ab7-4acb-acf7-6fc96f359a64" Nov 02 01:24:20.259606 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:20.259591 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:20.259730 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:20.259708 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgbsx" podUID="85784876-9397-4bec-8ade-878a15c09f40" Nov 02 01:24:20.259783 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:20.259770 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:20.259867 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:20.259851 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7szx6" podUID="79f97db3-814b-4849-ab8d-305dbcd3449c" Nov 02 01:24:20.432442 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:20.432413 2567 generic.go:356] "Generic (PLEG): container finished" podID="9b91ad19-9236-4395-9e54-09470929bf35" containerID="bc228dde72cff8ba3dac2557a0acf99404a1244cf560011650a2df2584335c00" exitCode=0 Nov 02 01:24:20.432963 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:20.432468 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mgjgw" event={"ID":"9b91ad19-9236-4395-9e54-09470929bf35","Type":"ContainerDied","Data":"bc228dde72cff8ba3dac2557a0acf99404a1244cf560011650a2df2584335c00"} Nov 02 01:24:21.437617 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:21.437577 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" event={"ID":"b083c3f1-9c74-4313-a939-ff8aa5769c8d","Type":"ContainerStarted","Data":"f612cba4d8cc7494994db307e641b5825bf0da9da163ddd5ca5a55f96d73b203"} Nov 02 01:24:21.440627 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:21.440604 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mgjgw" event={"ID":"9b91ad19-9236-4395-9e54-09470929bf35","Type":"ContainerStarted","Data":"0c2e118e067670121d944e232813534834fa65b669b93ef3be75f34364327a18"} Nov 02 01:24:21.562779 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:21.562731 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mgjgw" podStartSLOduration=8.960899959 podStartE2EDuration="21.562717122s" totalImagesPullingTime="12.601817163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:21.562205318 +0000 UTC m=+21.724614260" watchObservedRunningTime="2026-11-02 01:24:21.562717122 +0000 UTC m=+21.725126063" Nov 02 01:24:22.258323 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:22.258288 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:22.258323 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:22.258322 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:22.258533 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:22.258288 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:22.258533 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:22.258411 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flwvx" podUID="d628518e-2ab7-4acb-acf7-6fc96f359a64" Nov 02 01:24:22.258533 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:22.258451 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgbsx" podUID="85784876-9397-4bec-8ade-878a15c09f40" Nov 02 01:24:22.258533 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:22.258527 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7szx6" podUID="79f97db3-814b-4849-ab8d-305dbcd3449c" Nov 02 01:24:23.451927 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:23.451824 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" event={"ID":"b083c3f1-9c74-4313-a939-ff8aa5769c8d","Type":"ContainerStarted","Data":"1a2abfa6b79f5f9be3e81d06e4d1445b8d199e7e7f0d1001a640231f6b6f6a8a"} Nov 02 01:24:23.452342 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:23.452082 2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:23.452342 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:23.452152 2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:23.467389 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:23.467371 2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:23.468001 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:23.467987 2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:23.729073 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:23.728979 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" podStartSLOduration=7.775054925 podStartE2EDuration="23.72896649s" totalImagesPullingTime="15.953911565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 01:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:23.679257479 +0000 UTC m=+23.841666421" watchObservedRunningTime="2026-11-02 01:24:23.72896649 +0000 UTC m=+23.891375433" Nov 02 01:24:24.258387 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:24.258356 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:24.258387 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:24.258374 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:24.258730 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:24.258459 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:24.258730 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:24.258475 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgbsx" podUID="85784876-9397-4bec-8ade-878a15c09f40" Nov 02 01:24:24.258730 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:24.258533 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7szx6" podUID="79f97db3-814b-4849-ab8d-305dbcd3449c" Nov 02 01:24:24.258730 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:24.258589 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flwvx" podUID="d628518e-2ab7-4acb-acf7-6fc96f359a64" Nov 02 01:24:24.454357 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:24.454328 2567 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 01:24:25.456950 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:25.456920 2567 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 01:24:25.931500 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:25.931465 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7szx6"] Nov 02 01:24:25.931655 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:25.931585 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:25.931719 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:25.931694 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7szx6" podUID="79f97db3-814b-4849-ab8d-305dbcd3449c" Nov 02 01:24:25.934849 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:25.934816 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kgbsx"] Nov 02 01:24:25.934956 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:25.934894 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:25.935026 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:25.934972 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgbsx" podUID="85784876-9397-4bec-8ade-878a15c09f40" Nov 02 01:24:25.981101 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:25.981068 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-flwvx"] Nov 02 01:24:25.981214 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:25.981166 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:25.981265 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:25.981249 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flwvx" podUID="d628518e-2ab7-4acb-acf7-6fc96f359a64" Nov 02 01:24:26.078190 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:26.078165 2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:27.258351 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:27.258315 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:27.258741 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:27.258315 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:27.258741 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:27.258436 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7szx6" podUID="79f97db3-814b-4849-ab8d-305dbcd3449c" Nov 02 01:24:27.258741 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:27.258335 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:27.258741 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:27.258479 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgbsx" podUID="85784876-9397-4bec-8ade-878a15c09f40" Nov 02 01:24:27.258741 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:27.258608 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flwvx" podUID="d628518e-2ab7-4acb-acf7-6fc96f359a64" Nov 02 01:24:29.258859 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:29.258825 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:29.258859 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:29.258855 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:29.259280 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:29.258933 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:29.259280 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:29.258953 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7szx6" podUID="79f97db3-814b-4849-ab8d-305dbcd3449c" Nov 02 01:24:29.259280 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:29.259004 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgbsx" podUID="85784876-9397-4bec-8ade-878a15c09f40" Nov 02 01:24:29.259280 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:29.259079 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flwvx" podUID="d628518e-2ab7-4acb-acf7-6fc96f359a64" Nov 02 01:24:31.258653 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.258475 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:31.259035 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.258536 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:31.259035 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:31.258751 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7szx6" podUID="79f97db3-814b-4849-ab8d-305dbcd3449c" Nov 02 01:24:31.259035 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.258544 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:31.259035 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:31.258825 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kgbsx" podUID="85784876-9397-4bec-8ade-878a15c09f40" Nov 02 01:24:31.259035 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:31.258944 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-flwvx" podUID="d628518e-2ab7-4acb-acf7-6fc96f359a64" Nov 02 01:24:31.674529 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.674498 2567 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeReady" Nov 02 01:24:31.674702 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.674614 2567 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 01:24:31.764324 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.764290 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zlq59"] Nov 02 01:24:31.778379 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.778351 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zlq59" Nov 02 01:24:31.790317 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.790293 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-v8sdv"] Nov 02 01:24:31.799372 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.799354 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xmlts"] Nov 02 01:24:31.799486 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.799473 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-v8sdv" Nov 02 01:24:31.802317 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.802299 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz"] Nov 02 01:24:31.802431 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.802416 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xmlts" Nov 02 01:24:31.804693 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.804610 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-ntpqz"] Nov 02 01:24:31.805801 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.805787 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:31.807946 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.807929 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69469b6ffb-qmvll"] Nov 02 01:24:31.808052 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.808038 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:31.809576 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.809562 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tgsq4"] Nov 02 01:24:31.809704 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.809689 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:31.813952 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.813932 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d9wwq"] Nov 02 01:24:31.814061 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.814010 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:31.821683 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.821651 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6cb67dcd4c-zk2vs"] Nov 02 01:24:31.821785 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.821753 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d9wwq" Nov 02 01:24:31.826806 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.826782 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq"] Nov 02 01:24:31.826908 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.826869 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:31.830511 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.830323 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9"] Nov 02 01:24:31.830511 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.830424 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:31.835842 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.835825 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk"] Nov 02 01:24:31.835967 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.835951 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" Nov 02 01:24:31.839712 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.839413 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 01:24:31.839712 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.839590 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 01:24:31.840341 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.840323 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5954c6588f-7h92t"] Nov 02 01:24:31.840450 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.840434 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" Nov 02 01:24:31.845663 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.845647 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-58fr4"] Nov 02 01:24:31.845771 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.845757 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:31.850137 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.850111 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-frfx9"] Nov 02 01:24:31.850230 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.850166 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-58fr4" Nov 02 01:24:31.855685 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.855633 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9sgpk"] Nov 02 01:24:31.855795 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.855778 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:31.858152 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.858133 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 01:24:31.860775 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.860761 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qmrv6"] Nov 02 01:24:31.860886 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.860872 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9sgpk" Nov 02 01:24:31.867033 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.867013 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zlq59"] Nov 02 01:24:31.867126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.867053 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xmlts"] Nov 02 01:24:31.867126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.867081 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-v8sdv"] Nov 02 01:24:31.867126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.867101 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz"] Nov 02 01:24:31.867126 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.867120 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d9wwq"] Nov 02 01:24:31.867321 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.867180 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:31.873980 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.873953 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69469b6ffb-qmvll"] Nov 02 01:24:31.878290 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.878222 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk"] Nov 02 01:24:31.882240 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.882220 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9"] Nov 02 01:24:31.884929 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.884907 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq"] Nov 02 01:24:31.889125 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.889105 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-frfx9"] Nov 02 01:24:31.892131 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.892109 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9sgpk"] Nov 02 01:24:31.893089 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.893065 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5954c6588f-7h92t"] Nov 02 01:24:31.893660 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.893641 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qmrv6"] Nov 02 01:24:31.910418 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.910393 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6cb67dcd4c-zk2vs"] Nov 02 01:24:31.912851 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.912826 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-ntpqz"] Nov 02 01:24:31.927975 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.927930 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxmrq\" (UniqueName: \"kubernetes.io/projected/363dffa0-70cf-4fe0-996f-3c4d2f3b1a50-kube-api-access-bxmrq\") pod \"volume-data-source-validator-58c9f5f46d-d9wwq\" (UID: \"363dffa0-70cf-4fe0-996f-3c4d2f3b1a50\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d9wwq" Nov 02 01:24:31.927975 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.927958 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-default-certificate\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:31.928133 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.927978 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91cb4fd5-ed27-4a79-b172-be80b203bf0f-serving-cert\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:31.928133 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.927995 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-trusted-ca\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:31.928133 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.928113 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c2b846b-5c8b-4982-abc9-809fdbffe82d-image-registry-private-configuration\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:31.928276 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.928155 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-bound-sa-token\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:31.928276 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.928186 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq7hj\" (UniqueName: \"kubernetes.io/projected/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-kube-api-access-rq7hj\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:31.928276 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.928256 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-installation-pull-secrets\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:31.928384 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.928298 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-image-registry-private-configuration\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:31.928384 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.928316 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjz2b\" (UniqueName: \"kubernetes.io/projected/8618e1e8-4d71-4f7a-b2a1-1b950190f110-kube-api-access-wjz2b\") pod \"kas-connection-checker-685895c7c9-xmlts\" (UID: \"8618e1e8-4d71-4f7a-b2a1-1b950190f110\") " pod="kube-system/kas-connection-checker-685895c7c9-xmlts" Nov 02 01:24:31.928384 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.928335 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-bound-sa-token\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:31.928384 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.928352 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:31.928518 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.928411 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c285da3-2e11-498d-94f1-76c0ce604553-trusted-ca\") pod \"console-operator-7cd56b8dbc-frfx9\" (UID: \"7c285da3-2e11-498d-94f1-76c0ce604553\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:31.928518 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.928436 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-metrics-tls\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:31.928518 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.928455 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c2b846b-5c8b-4982-abc9-809fdbffe82d-ca-trust-extracted\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:31.928518 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.928470 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e79ec42c-3696-40e7-94de-57aad9a9f92c-cert\") pod \"ingress-canary-9sgpk\" (UID: \"e79ec42c-3696-40e7-94de-57aad9a9f92c\") " pod="openshift-ingress-canary/ingress-canary-9sgpk" Nov 02 01:24:31.928518 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.928485 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp982\" (UniqueName: \"kubernetes.io/projected/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-kube-api-access-xp982\") pod \"cluster-monitoring-operator-884576c57-6sdmz\" (UID: \"59ccadf3-202d-481d-8c8b-c4fa86c92b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:31.928518 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.928509 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z8mm\" (UniqueName: \"kubernetes.io/projected/2e031e5c-cb8a-42b9-b334-1cbe61a9c59f-kube-api-access-6z8mm\") pod \"kas-connection-checker-685895c7c9-zlq59\" (UID: \"2e031e5c-cb8a-42b9-b334-1cbe61a9c59f\") " pod="kube-system/kas-connection-checker-685895c7c9-zlq59" Nov 02 01:24:31.928788 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.928538 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-nginx-conf\") pod \"networking-console-plugin-86c478bb88-ntpqz\" (UID: \"fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:31.928788 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.928554 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:31.928952 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.928931 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6sdmz\" (UID: \"59ccadf3-202d-481d-8c8b-c4fa86c92b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:31.928989 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.928960 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44rfr\" (UniqueName: \"kubernetes.io/projected/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-kube-api-access-44rfr\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:31.928989 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.928978 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:31.929082 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.928993 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-stats-auth\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:31.929082 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929063 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8wfd\" (UniqueName: \"kubernetes.io/projected/7c285da3-2e11-498d-94f1-76c0ce604553-kube-api-access-v8wfd\") pod \"console-operator-7cd56b8dbc-frfx9\" (UID: \"7c285da3-2e11-498d-94f1-76c0ce604553\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:31.929173 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929094 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6sdmz\" (UID: \"59ccadf3-202d-481d-8c8b-c4fa86c92b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:31.929173 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929122 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:31.929259 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929172 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c2b846b-5c8b-4982-abc9-809fdbffe82d-trusted-ca\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:31.929259 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929189 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c285da3-2e11-498d-94f1-76c0ce604553-serving-cert\") pod \"console-operator-7cd56b8dbc-frfx9\" (UID: \"7c285da3-2e11-498d-94f1-76c0ce604553\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:31.929259 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929204 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4abd5311-4a92-4ac5-8aa0-779dd8774577-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtll9\" (UID: \"4abd5311-4a92-4ac5-8aa0-779dd8774577\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" Nov 02 01:24:31.929395 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929264 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:31.929395 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929296 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ntpqz\" (UID: \"fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:31.929395 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929323 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-certificates\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:31.929395 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929348 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35ae4588-f166-4563-aae7-b34b5b053d0f-serving-cert\") pod \"service-ca-operator-544d749c8c-868jk\" (UID: \"35ae4588-f166-4563-aae7-b34b5b053d0f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" Nov 02 01:24:31.929395 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929373 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/35ae4588-f166-4563-aae7-b34b5b053d0f-config\") pod \"service-ca-operator-544d749c8c-868jk\" (UID: \"35ae4588-f166-4563-aae7-b34b5b053d0f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" Nov 02 01:24:31.929563 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929401 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91cb4fd5-ed27-4a79-b172-be80b203bf0f-tmp\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:31.929563 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929417 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdnrv\" (UniqueName: \"kubernetes.io/projected/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-kube-api-access-kdnrv\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:31.929563 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929433 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c2b846b-5c8b-4982-abc9-809fdbffe82d-installation-pull-secrets\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:31.929563 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929457 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:31.929563 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929525 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpwzf\" (UniqueName: \"kubernetes.io/projected/4abd5311-4a92-4ac5-8aa0-779dd8774577-kube-api-access-tpwzf\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtll9\" (UID: \"4abd5311-4a92-4ac5-8aa0-779dd8774577\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" Nov 02 01:24:31.929813 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929566 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91cb4fd5-ed27-4a79-b172-be80b203bf0f-service-ca-bundle\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:31.929813 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929586 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c285da3-2e11-498d-94f1-76c0ce604553-config\") pod \"console-operator-7cd56b8dbc-frfx9\" (UID: \"7c285da3-2e11-498d-94f1-76c0ce604553\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:31.929813 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929633 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4abd5311-4a92-4ac5-8aa0-779dd8774577-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtll9\" (UID: \"4abd5311-4a92-4ac5-8aa0-779dd8774577\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" Nov 02 01:24:31.929813 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929716 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tghz\" (UniqueName: \"kubernetes.io/projected/e79ec42c-3696-40e7-94de-57aad9a9f92c-kube-api-access-4tghz\") pod \"ingress-canary-9sgpk\" (UID: \"e79ec42c-3696-40e7-94de-57aad9a9f92c\") " pod="openshift-ingress-canary/ingress-canary-9sgpk" Nov 02 01:24:31.929813 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929742 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-config-volume\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:31.929813 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929771 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnz7p\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-kube-api-access-tnz7p\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:31.929813 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929799 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91cb4fd5-ed27-4a79-b172-be80b203bf0f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:31.930031 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929841 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-tmp-dir\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:31.930031 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929867 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvshf\" (UniqueName: \"kubernetes.io/projected/2c8cbfce-aaf5-4362-930e-b15539b2dbc4-kube-api-access-jvshf\") pod \"network-check-source-6f78d59f99-58fr4\" (UID: \"2c8cbfce-aaf5-4362-930e-b15539b2dbc4\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-58fr4" Nov 02 01:24:31.930031 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929897 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/91cb4fd5-ed27-4a79-b172-be80b203bf0f-snapshots\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:31.930031 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929918 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s6ct\" (UniqueName: \"kubernetes.io/projected/35ae4588-f166-4563-aae7-b34b5b053d0f-kube-api-access-6s6ct\") pod \"service-ca-operator-544d749c8c-868jk\" (UID: \"35ae4588-f166-4563-aae7-b34b5b053d0f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" Nov 02 01:24:31.930031 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929943 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh798\" (UniqueName: \"kubernetes.io/projected/eb8ad2e6-7e4f-4c0a-9ea3-99ed98344ba6-kube-api-access-xh798\") pod \"kas-connection-checker-685895c7c9-v8sdv\" (UID: \"eb8ad2e6-7e4f-4c0a-9ea3-99ed98344ba6\") " pod="kube-system/kas-connection-checker-685895c7c9-v8sdv" Nov 02 01:24:31.930031 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.929982 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-ca-trust-extracted\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:31.930031 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.930002 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-certificates\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:31.930031 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.930021 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4q2q\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-kube-api-access-r4q2q\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:31.930271 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.930041 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-862mp\" (UniqueName: \"kubernetes.io/projected/91cb4fd5-ed27-4a79-b172-be80b203bf0f-kube-api-access-862mp\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:31.934931 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.934910 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:24:31.934931 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.934926 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 01:24:31.935057 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.934915 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 01:24:31.935057 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.934912 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 01:24:31.935132 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935070 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 01:24:31.935185 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935140 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 01:24:31.936102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935508 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 01:24:31.936102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935202 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 01:24:31.936102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935263 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 01:24:31.936102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935542 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 01:24:31.936102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935290 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:24:31.936102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935293 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 01:24:31.936102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935306 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 01:24:31.936102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935322 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:24:31.936102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935614 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 01:24:31.936102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935374 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 01:24:31.936102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935412 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 01:24:31.936102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935424 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:24:31.936102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935429 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 01:24:31.936102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935486 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:24:31.936102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935507 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 01:24:31.936102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935228 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 01:24:31.936102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935542 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 01:24:31.936102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935290 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 01:24:31.936102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935357 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 01:24:31.936102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.935975 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 01:24:31.941716 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.941697 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tgsq4"] Nov 02 01:24:31.950902 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.950841 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 01:24:31.950902 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.950894 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 01:24:31.990462 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.990429 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 01:24:31.990462 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:31.990456 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 01:24:32.000725 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.000704 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 01:24:32.000932 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.000913 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 01:24:32.000932 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.000925 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 01:24:32.001068 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.000962 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 01:24:32.001068 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.000969 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 01:24:32.001068 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.000997 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 01:24:32.001068 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.001010 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 01:24:32.001068 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.001021 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 01:24:32.001068 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.001046 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jvpgw\"" Nov 02 01:24:32.001068 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.001059 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 01:24:32.001504 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.001075 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 01:24:32.001504 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.001084 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 01:24:32.001504 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.001085 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 01:24:32.001504 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.001103 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 01:24:32.001504 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.001104 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-vkh6q\"" Nov 02 01:24:32.001504 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.001131 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 01:24:32.001504 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.001164 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tl75j\"" Nov 02 01:24:32.001504 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.001247 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 01:24:32.001504 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.001267 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q5j6l\"" Nov 02 01:24:32.001805 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.001556 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 01:24:32.001805 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.001710 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-58fr4"] Nov 02 01:24:32.030535 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.030511 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4abd5311-4a92-4ac5-8aa0-779dd8774577-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtll9\" (UID: \"4abd5311-4a92-4ac5-8aa0-779dd8774577\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" Nov 02 01:24:32.030611 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.030542 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tghz\" (UniqueName: \"kubernetes.io/projected/e79ec42c-3696-40e7-94de-57aad9a9f92c-kube-api-access-4tghz\") pod \"ingress-canary-9sgpk\" (UID: \"e79ec42c-3696-40e7-94de-57aad9a9f92c\") " pod="openshift-ingress-canary/ingress-canary-9sgpk" Nov 02 01:24:32.030611 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.030559 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-config-volume\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:32.030611 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.030585 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnz7p\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-kube-api-access-tnz7p\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:32.030611 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.030608 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91cb4fd5-ed27-4a79-b172-be80b203bf0f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:32.030834 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.030781 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-tmp-dir\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:32.030834 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.030810 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvshf\" (UniqueName: \"kubernetes.io/projected/2c8cbfce-aaf5-4362-930e-b15539b2dbc4-kube-api-access-jvshf\") pod \"network-check-source-6f78d59f99-58fr4\" (UID: \"2c8cbfce-aaf5-4362-930e-b15539b2dbc4\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-58fr4" Nov 02 01:24:32.030834 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.030831 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/91cb4fd5-ed27-4a79-b172-be80b203bf0f-snapshots\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:32.030981 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.030855 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s6ct\" (UniqueName: \"kubernetes.io/projected/35ae4588-f166-4563-aae7-b34b5b053d0f-kube-api-access-6s6ct\") pod \"service-ca-operator-544d749c8c-868jk\" (UID: \"35ae4588-f166-4563-aae7-b34b5b053d0f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" Nov 02 01:24:32.030981 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.030878 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh798\" (UniqueName: \"kubernetes.io/projected/eb8ad2e6-7e4f-4c0a-9ea3-99ed98344ba6-kube-api-access-xh798\") pod \"kas-connection-checker-685895c7c9-v8sdv\" (UID: \"eb8ad2e6-7e4f-4c0a-9ea3-99ed98344ba6\") " pod="kube-system/kas-connection-checker-685895c7c9-v8sdv" Nov 02 01:24:32.030981 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.030906 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-ca-trust-extracted\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:32.030981 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.030932 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-certificates\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:32.030981 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.030953 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4q2q\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-kube-api-access-r4q2q\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:32.031215 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.030977 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-862mp\" (UniqueName: \"kubernetes.io/projected/91cb4fd5-ed27-4a79-b172-be80b203bf0f-kube-api-access-862mp\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:32.031215 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031011 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxmrq\" (UniqueName: \"kubernetes.io/projected/363dffa0-70cf-4fe0-996f-3c4d2f3b1a50-kube-api-access-bxmrq\") pod \"volume-data-source-validator-58c9f5f46d-d9wwq\" (UID: \"363dffa0-70cf-4fe0-996f-3c4d2f3b1a50\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d9wwq" Nov 02 01:24:32.031215 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031037 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-default-certificate\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:32.031215 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031064 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91cb4fd5-ed27-4a79-b172-be80b203bf0f-serving-cert\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:32.031215 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031091 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-trusted-ca\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:32.031215 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031117 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c2b846b-5c8b-4982-abc9-809fdbffe82d-image-registry-private-configuration\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:32.031215 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031124 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-tmp-dir\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:32.031215 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031142 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-bound-sa-token\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:32.031215 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031167 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq7hj\" (UniqueName: \"kubernetes.io/projected/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-kube-api-access-rq7hj\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:32.031215 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031217 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-installation-pull-secrets\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:32.031756 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031259 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-image-registry-private-configuration\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:32.031756 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031274 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-config-volume\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:32.031756 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031284 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjz2b\" (UniqueName: \"kubernetes.io/projected/8618e1e8-4d71-4f7a-b2a1-1b950190f110-kube-api-access-wjz2b\") pod \"kas-connection-checker-685895c7c9-xmlts\" (UID: \"8618e1e8-4d71-4f7a-b2a1-1b950190f110\") " pod="kube-system/kas-connection-checker-685895c7c9-xmlts" Nov 02 01:24:32.031756 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031309 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-bound-sa-token\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:32.031756 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031348 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:32.031756 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031381 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c285da3-2e11-498d-94f1-76c0ce604553-trusted-ca\") pod \"console-operator-7cd56b8dbc-frfx9\" (UID: \"7c285da3-2e11-498d-94f1-76c0ce604553\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:32.031756 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031413 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-metrics-tls\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:32.031756 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031447 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c2b846b-5c8b-4982-abc9-809fdbffe82d-ca-trust-extracted\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:32.031756 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031459 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/91cb4fd5-ed27-4a79-b172-be80b203bf0f-snapshots\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:32.031756 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031473 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e79ec42c-3696-40e7-94de-57aad9a9f92c-cert\") pod \"ingress-canary-9sgpk\" (UID: \"e79ec42c-3696-40e7-94de-57aad9a9f92c\") " pod="openshift-ingress-canary/ingress-canary-9sgpk" Nov 02 01:24:32.031756 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031501 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp982\" (UniqueName: \"kubernetes.io/projected/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-kube-api-access-xp982\") pod \"cluster-monitoring-operator-884576c57-6sdmz\" (UID: \"59ccadf3-202d-481d-8c8b-c4fa86c92b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:32.031756 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031527 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z8mm\" (UniqueName: \"kubernetes.io/projected/2e031e5c-cb8a-42b9-b334-1cbe61a9c59f-kube-api-access-6z8mm\") pod \"kas-connection-checker-685895c7c9-zlq59\" (UID: \"2e031e5c-cb8a-42b9-b334-1cbe61a9c59f\") " pod="kube-system/kas-connection-checker-685895c7c9-zlq59" Nov 02 01:24:32.031756 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031535 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91cb4fd5-ed27-4a79-b172-be80b203bf0f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:32.031756 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031552 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-nginx-conf\") pod \"networking-console-plugin-86c478bb88-ntpqz\" (UID: \"fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:32.031756 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.031582 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:24:32.031756 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031605 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:32.032512 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.031648 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-metrics-tls podName:da5a0b9c-57f5-4741-94f3-fb9fe7b7e699 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:32.531630352 +0000 UTC m=+32.694039272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-metrics-tls") pod "dns-default-qmrv6" (UID: "da5a0b9c-57f5-4741-94f3-fb9fe7b7e699") : secret "dns-default-metrics-tls" not found Nov 02 01:24:32.032512 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031701 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6sdmz\" (UID: \"59ccadf3-202d-481d-8c8b-c4fa86c92b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:32.032512 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031740 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44rfr\" (UniqueName: \"kubernetes.io/projected/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-kube-api-access-44rfr\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:32.032512 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031769 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:32.032512 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031817 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-stats-auth\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:32.032512 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031845 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8wfd\" (UniqueName: \"kubernetes.io/projected/7c285da3-2e11-498d-94f1-76c0ce604553-kube-api-access-v8wfd\") pod \"console-operator-7cd56b8dbc-frfx9\" (UID: \"7c285da3-2e11-498d-94f1-76c0ce604553\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:32.032512 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031906 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6sdmz\" (UID: \"59ccadf3-202d-481d-8c8b-c4fa86c92b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:32.032512 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.031929 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 01:24:32.032512 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031936 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:32.032512 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.032217 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-ca-trust-extracted\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:32.032512 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.031943 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5954c6588f-7h92t: secret "image-registry-tls" not found Nov 02 01:24:32.032512 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.031981 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle podName:bdebb5b6-76a2-48c8-810d-6f80549a4f3c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:32.531960843 +0000 UTC m=+32.694369766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle") pod "router-default-6cb67dcd4c-zk2vs" (UID: "bdebb5b6-76a2-48c8-810d-6f80549a4f3c") : configmap references non-existent config key: service-ca.crt Nov 02 01:24:32.032512 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.032229 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 01:24:32.032512 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.032293 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69469b6ffb-qmvll: secret "image-registry-tls" not found Nov 02 01:24:32.032512 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.032346 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls podName:7c2b846b-5c8b-4982-abc9-809fdbffe82d nodeName:}" failed. No retries permitted until 2026-11-02 01:24:32.532332086 +0000 UTC m=+32.694741009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls") pod "image-registry-5954c6588f-7h92t" (UID: "7c2b846b-5c8b-4982-abc9-809fdbffe82d") : secret "image-registry-tls" not found Nov 02 01:24:32.033239 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.032361 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls podName:d5c07387-ae45-48d3-8b9d-8f328bdfd63c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:32.532352875 +0000 UTC m=+32.694761795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls") pod "image-registry-69469b6ffb-qmvll" (UID: "d5c07387-ae45-48d3-8b9d-8f328bdfd63c") : secret "image-registry-tls" not found Nov 02 01:24:32.033239 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.032378 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c2b846b-5c8b-4982-abc9-809fdbffe82d-trusted-ca\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:32.033239 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.032405 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-certificates\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:32.033239 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.032579 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-nginx-conf\") pod \"networking-console-plugin-86c478bb88-ntpqz\" (UID: \"fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:32.033239 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.032406 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c285da3-2e11-498d-94f1-76c0ce604553-serving-cert\") pod \"console-operator-7cd56b8dbc-frfx9\" (UID: \"7c285da3-2e11-498d-94f1-76c0ce604553\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:32.033239 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.032860 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4abd5311-4a92-4ac5-8aa0-779dd8774577-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtll9\" (UID: \"4abd5311-4a92-4ac5-8aa0-779dd8774577\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" Nov 02 01:24:32.033239 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.032892 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:32.033239 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.032927 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ntpqz\" (UID: \"fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:32.033239 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.032962 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-certificates\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:32.033239 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.032991 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35ae4588-f166-4563-aae7-b34b5b053d0f-serving-cert\") pod \"service-ca-operator-544d749c8c-868jk\" (UID: \"35ae4588-f166-4563-aae7-b34b5b053d0f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" Nov 02 01:24:32.033239 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.033023 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/35ae4588-f166-4563-aae7-b34b5b053d0f-config\") pod \"service-ca-operator-544d749c8c-868jk\" (UID: \"35ae4588-f166-4563-aae7-b34b5b053d0f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" Nov 02 01:24:32.033239 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.033054 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91cb4fd5-ed27-4a79-b172-be80b203bf0f-tmp\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:32.033239 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.033081 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdnrv\" (UniqueName: \"kubernetes.io/projected/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-kube-api-access-kdnrv\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:32.033239 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.033112 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c2b846b-5c8b-4982-abc9-809fdbffe82d-installation-pull-secrets\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:32.033239 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.033149 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 01:24:32.033239 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.033181 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c2b846b-5c8b-4982-abc9-809fdbffe82d-trusted-ca\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:32.034010 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.033200 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert podName:fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:32.533185688 +0000 UTC m=+32.695594610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ntpqz" (UID: "fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41") : secret "networking-console-plugin-cert" not found Nov 02 01:24:32.034010 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.033229 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 01:24:32.034010 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.033288 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls podName:b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b nodeName:}" failed. No retries permitted until 2026-11-02 01:24:32.533273473 +0000 UTC m=+32.695682402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-brlzq" (UID: "b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b") : secret "samples-operator-tls" not found Nov 02 01:24:32.034010 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.033505 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-trusted-ca\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:32.034010 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.033804 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-certificates\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:32.034010 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.034001 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c285da3-2e11-498d-94f1-76c0ce604553-trusted-ca\") pod \"console-operator-7cd56b8dbc-frfx9\" (UID: \"7c285da3-2e11-498d-94f1-76c0ce604553\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:32.034329 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.034119 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91cb4fd5-ed27-4a79-b172-be80b203bf0f-tmp\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:32.034576 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.034552 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4abd5311-4a92-4ac5-8aa0-779dd8774577-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtll9\" (UID: \"4abd5311-4a92-4ac5-8aa0-779dd8774577\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" Nov 02 01:24:32.034731 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.034696 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/35ae4588-f166-4563-aae7-b34b5b053d0f-config\") pod \"service-ca-operator-544d749c8c-868jk\" (UID: \"35ae4588-f166-4563-aae7-b34b5b053d0f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" Nov 02 01:24:32.036765 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.033154 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:32.036765 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.034854 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpwzf\" (UniqueName: \"kubernetes.io/projected/4abd5311-4a92-4ac5-8aa0-779dd8774577-kube-api-access-tpwzf\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtll9\" (UID: \"4abd5311-4a92-4ac5-8aa0-779dd8774577\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" Nov 02 01:24:32.036765 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.034896 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91cb4fd5-ed27-4a79-b172-be80b203bf0f-service-ca-bundle\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:32.036765 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.034931 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c285da3-2e11-498d-94f1-76c0ce604553-config\") pod \"console-operator-7cd56b8dbc-frfx9\" (UID: \"7c285da3-2e11-498d-94f1-76c0ce604553\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:32.036765 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.035211 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:32.036765 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.035606 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c285da3-2e11-498d-94f1-76c0ce604553-config\") pod \"console-operator-7cd56b8dbc-frfx9\" (UID: \"7c285da3-2e11-498d-94f1-76c0ce604553\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:32.036765 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.035853 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 01:24:32.036765 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.035907 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-cluster-monitoring-operator-tls podName:59ccadf3-202d-481d-8c8b-c4fa86c92b8b nodeName:}" failed. No retries permitted until 2026-11-02 01:24:32.535892617 +0000 UTC m=+32.698301551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6sdmz" (UID: "59ccadf3-202d-481d-8c8b-c4fa86c92b8b") : secret "cluster-monitoring-operator-tls" not found Nov 02 01:24:32.036765 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.036145 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91cb4fd5-ed27-4a79-b172-be80b203bf0f-service-ca-bundle\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:32.036765 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.036422 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91cb4fd5-ed27-4a79-b172-be80b203bf0f-serving-cert\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:32.036765 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.031935 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c2b846b-5c8b-4982-abc9-809fdbffe82d-ca-trust-extracted\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:32.036765 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.036501 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 01:24:32.036765 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.036559 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs podName:bdebb5b6-76a2-48c8-810d-6f80549a4f3c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:32.536543912 +0000 UTC m=+32.698952846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs") pod "router-default-6cb67dcd4c-zk2vs" (UID: "bdebb5b6-76a2-48c8-810d-6f80549a4f3c") : secret "router-metrics-certs-default" not found Nov 02 01:24:32.036765 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.036584 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6sdmz\" (UID: \"59ccadf3-202d-481d-8c8b-c4fa86c92b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:32.036765 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.036647 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-stats-auth\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:32.037577 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.036681 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35ae4588-f166-4563-aae7-b34b5b053d0f-serving-cert\") pod \"service-ca-operator-544d749c8c-868jk\" (UID: \"35ae4588-f166-4563-aae7-b34b5b053d0f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" Nov 02 01:24:32.037577 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.037274 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c2b846b-5c8b-4982-abc9-809fdbffe82d-image-registry-private-configuration\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:32.037577 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.037327 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-installation-pull-secrets\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:32.037882 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.037856 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-default-certificate\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:32.037882 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.037872 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e79ec42c-3696-40e7-94de-57aad9a9f92c-cert\") pod \"ingress-canary-9sgpk\" (UID: \"e79ec42c-3696-40e7-94de-57aad9a9f92c\") " pod="openshift-ingress-canary/ingress-canary-9sgpk" Nov 02 01:24:32.038230 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.038208 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4abd5311-4a92-4ac5-8aa0-779dd8774577-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtll9\" (UID: \"4abd5311-4a92-4ac5-8aa0-779dd8774577\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" Nov 02 01:24:32.038313 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.038295 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-image-registry-private-configuration\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:32.038539 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.038521 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c2b846b-5c8b-4982-abc9-809fdbffe82d-installation-pull-secrets\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:32.038597 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.038564 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c285da3-2e11-498d-94f1-76c0ce604553-serving-cert\") pod \"console-operator-7cd56b8dbc-frfx9\" (UID: \"7c285da3-2e11-498d-94f1-76c0ce604553\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:32.094653 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.094610 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tghz\" (UniqueName: \"kubernetes.io/projected/e79ec42c-3696-40e7-94de-57aad9a9f92c-kube-api-access-4tghz\") pod \"ingress-canary-9sgpk\" (UID: \"e79ec42c-3696-40e7-94de-57aad9a9f92c\") " pod="openshift-ingress-canary/ingress-canary-9sgpk" Nov 02 01:24:32.096807 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.096783 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s6ct\" (UniqueName: \"kubernetes.io/projected/35ae4588-f166-4563-aae7-b34b5b053d0f-kube-api-access-6s6ct\") pod \"service-ca-operator-544d749c8c-868jk\" (UID: \"35ae4588-f166-4563-aae7-b34b5b053d0f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" Nov 02 01:24:32.096919 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.096804 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjz2b\" (UniqueName: \"kubernetes.io/projected/8618e1e8-4d71-4f7a-b2a1-1b950190f110-kube-api-access-wjz2b\") pod \"kas-connection-checker-685895c7c9-xmlts\" (UID: \"8618e1e8-4d71-4f7a-b2a1-1b950190f110\") " pod="kube-system/kas-connection-checker-685895c7c9-xmlts" Nov 02 01:24:32.097104 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.097081 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44rfr\" (UniqueName: \"kubernetes.io/projected/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-kube-api-access-44rfr\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:32.097371 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.097348 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnz7p\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-kube-api-access-tnz7p\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:32.097839 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.097818 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4q2q\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-kube-api-access-r4q2q\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:32.103423 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.103396 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxmrq\" (UniqueName: \"kubernetes.io/projected/363dffa0-70cf-4fe0-996f-3c4d2f3b1a50-kube-api-access-bxmrq\") pod \"volume-data-source-validator-58c9f5f46d-d9wwq\" (UID: \"363dffa0-70cf-4fe0-996f-3c4d2f3b1a50\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d9wwq" Nov 02 01:24:32.105245 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.104048 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-862mp\" (UniqueName: \"kubernetes.io/projected/91cb4fd5-ed27-4a79-b172-be80b203bf0f-kube-api-access-862mp\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:32.105245 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.104267 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq7hj\" (UniqueName: \"kubernetes.io/projected/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-kube-api-access-rq7hj\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:32.105245 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.104323 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-bound-sa-token\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:32.105245 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.104630 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdnrv\" (UniqueName: \"kubernetes.io/projected/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-kube-api-access-kdnrv\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:32.105245 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.104632 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpwzf\" (UniqueName: \"kubernetes.io/projected/4abd5311-4a92-4ac5-8aa0-779dd8774577-kube-api-access-tpwzf\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtll9\" (UID: \"4abd5311-4a92-4ac5-8aa0-779dd8774577\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" Nov 02 01:24:32.105245 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.104782 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8wfd\" (UniqueName: \"kubernetes.io/projected/7c285da3-2e11-498d-94f1-76c0ce604553-kube-api-access-v8wfd\") pod \"console-operator-7cd56b8dbc-frfx9\" (UID: \"7c285da3-2e11-498d-94f1-76c0ce604553\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:32.105245 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.105031 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z8mm\" (UniqueName: \"kubernetes.io/projected/2e031e5c-cb8a-42b9-b334-1cbe61a9c59f-kube-api-access-6z8mm\") pod \"kas-connection-checker-685895c7c9-zlq59\" (UID: \"2e031e5c-cb8a-42b9-b334-1cbe61a9c59f\") " pod="kube-system/kas-connection-checker-685895c7c9-zlq59" Nov 02 01:24:32.105245 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.105211 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp982\" (UniqueName: \"kubernetes.io/projected/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-kube-api-access-xp982\") pod \"cluster-monitoring-operator-884576c57-6sdmz\" (UID: \"59ccadf3-202d-481d-8c8b-c4fa86c92b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:32.105575 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.105510 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvshf\" (UniqueName: \"kubernetes.io/projected/2c8cbfce-aaf5-4362-930e-b15539b2dbc4-kube-api-access-jvshf\") pod \"network-check-source-6f78d59f99-58fr4\" (UID: \"2c8cbfce-aaf5-4362-930e-b15539b2dbc4\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-58fr4" Nov 02 01:24:32.105962 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.105939 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh798\" (UniqueName: \"kubernetes.io/projected/eb8ad2e6-7e4f-4c0a-9ea3-99ed98344ba6-kube-api-access-xh798\") pod \"kas-connection-checker-685895c7c9-v8sdv\" (UID: \"eb8ad2e6-7e4f-4c0a-9ea3-99ed98344ba6\") " pod="kube-system/kas-connection-checker-685895c7c9-v8sdv" Nov 02 01:24:32.106183 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.106166 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-bound-sa-token\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:32.115305 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.115279 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-v8sdv" Nov 02 01:24:32.124637 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.124616 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xmlts" Nov 02 01:24:32.152491 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.152315 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:32.159295 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.159273 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d9wwq" Nov 02 01:24:32.181521 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.181432 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" Nov 02 01:24:32.185169 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.185110 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" Nov 02 01:24:32.237592 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.237325 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-58fr4" Nov 02 01:24:32.272368 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.272330 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:32.295270 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.295231 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9sgpk" Nov 02 01:24:32.396367 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.396343 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zlq59" Nov 02 01:24:32.535417 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.535376 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-v8sdv"] Nov 02 01:24:32.538968 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:32.538938 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8ad2e6_7e4f_4c0a_9ea3_99ed98344ba6.slice/crio-aef0f7510a57f5b33b937b4d573d2417f0dbf9f2f55ee57de1f762e2ea09b55d WatchSource:0}: Error finding container aef0f7510a57f5b33b937b4d573d2417f0dbf9f2f55ee57de1f762e2ea09b55d: Status 404 returned error can't find the container with id aef0f7510a57f5b33b937b4d573d2417f0dbf9f2f55ee57de1f762e2ea09b55d Nov 02 01:24:32.540156 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.540135 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:32.540250 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.540232 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:32.540313 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.540264 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-metrics-tls\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:32.540313 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.540282 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 01:24:32.540313 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.540296 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:32.540454 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.540325 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6sdmz\" (UID: \"59ccadf3-202d-481d-8c8b-c4fa86c92b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:32.540454 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.540344 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls podName:b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b nodeName:}" failed. No retries permitted until 2026-11-02 01:24:33.540323803 +0000 UTC m=+33.702732726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-brlzq" (UID: "b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b") : secret "samples-operator-tls" not found Nov 02 01:24:32.540454 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.540383 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:32.540454 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.540398 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 01:24:32.540454 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.540401 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 01:24:32.540454 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.540412 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:32.540454 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.540419 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5954c6588f-7h92t: secret "image-registry-tls" not found Nov 02 01:24:32.540454 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.540440 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle podName:bdebb5b6-76a2-48c8-810d-6f80549a4f3c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:33.540426117 +0000 UTC m=+33.702835057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle") pod "router-default-6cb67dcd4c-zk2vs" (UID: "bdebb5b6-76a2-48c8-810d-6f80549a4f3c") : configmap references non-existent config key: service-ca.crt Nov 02 01:24:32.540454 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.540397 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:24:32.540884 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.540466 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-cluster-monitoring-operator-tls podName:59ccadf3-202d-481d-8c8b-c4fa86c92b8b nodeName:}" failed. No retries permitted until 2026-11-02 01:24:33.540456533 +0000 UTC m=+33.702865454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6sdmz" (UID: "59ccadf3-202d-481d-8c8b-c4fa86c92b8b") : secret "cluster-monitoring-operator-tls" not found Nov 02 01:24:32.540884 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.540482 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 01:24:32.540884 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.540485 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls podName:7c2b846b-5c8b-4982-abc9-809fdbffe82d nodeName:}" failed. No retries permitted until 2026-11-02 01:24:33.540475096 +0000 UTC m=+33.702884019 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls") pod "image-registry-5954c6588f-7h92t" (UID: "7c2b846b-5c8b-4982-abc9-809fdbffe82d") : secret "image-registry-tls" not found Nov 02 01:24:32.540884 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.540489 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 01:24:32.540884 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.540506 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-metrics-tls podName:da5a0b9c-57f5-4741-94f3-fb9fe7b7e699 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:33.540497116 +0000 UTC m=+33.702906038 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-metrics-tls") pod "dns-default-qmrv6" (UID: "da5a0b9c-57f5-4741-94f3-fb9fe7b7e699") : secret "dns-default-metrics-tls" not found Nov 02 01:24:32.540884 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.540491 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69469b6ffb-qmvll: secret "image-registry-tls" not found Nov 02 01:24:32.540884 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.540522 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs podName:bdebb5b6-76a2-48c8-810d-6f80549a4f3c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:33.540513512 +0000 UTC m=+33.702922449 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs") pod "router-default-6cb67dcd4c-zk2vs" (UID: "bdebb5b6-76a2-48c8-810d-6f80549a4f3c") : secret "router-metrics-certs-default" not found Nov 02 01:24:32.540884 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.540542 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls podName:d5c07387-ae45-48d3-8b9d-8f328bdfd63c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:33.540534562 +0000 UTC m=+33.702943497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls") pod "image-registry-69469b6ffb-qmvll" (UID: "d5c07387-ae45-48d3-8b9d-8f328bdfd63c") : secret "image-registry-tls" not found Nov 02 01:24:32.540884 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.540561 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ntpqz\" (UID: \"fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:32.540884 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.540640 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 01:24:32.540884 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:32.540661 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert podName:fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:33.540654793 +0000 UTC m=+33.703063713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ntpqz" (UID: "fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41") : secret "networking-console-plugin-cert" not found Nov 02 01:24:32.551847 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.551818 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-tgsq4"] Nov 02 01:24:32.575814 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:32.575771 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cb4fd5_ed27_4a79_b172_be80b203bf0f.slice/crio-d1b449089fc1b096507ccc81210b0ab22d0eef7f4816aa2fdafe9b3542bdbd4a WatchSource:0}: Error finding container d1b449089fc1b096507ccc81210b0ab22d0eef7f4816aa2fdafe9b3542bdbd4a: Status 404 returned error can't find the container with id d1b449089fc1b096507ccc81210b0ab22d0eef7f4816aa2fdafe9b3542bdbd4a Nov 02 01:24:32.583782 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.583758 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d9wwq"] Nov 02 01:24:32.587122 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:32.587091 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363dffa0_70cf_4fe0_996f_3c4d2f3b1a50.slice/crio-e1fbaf835ff8f3a7565f0d4aefabd3fbdfac25aa4676e785fad3e60cc44e6a38 WatchSource:0}: Error finding container e1fbaf835ff8f3a7565f0d4aefabd3fbdfac25aa4676e785fad3e60cc44e6a38: Status 404 returned error can't find the container with id e1fbaf835ff8f3a7565f0d4aefabd3fbdfac25aa4676e785fad3e60cc44e6a38 Nov 02 01:24:32.590201 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.590137 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9"] Nov 02 01:24:32.593620 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:32.593589 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abd5311_4a92_4ac5_8aa0_779dd8774577.slice/crio-780e6524fb0d75e94a6e9d5e7000be249c6dc9f4a2fdae31a49b93c800814f62 WatchSource:0}: Error finding container 780e6524fb0d75e94a6e9d5e7000be249c6dc9f4a2fdae31a49b93c800814f62: Status 404 returned error can't find the container with id 780e6524fb0d75e94a6e9d5e7000be249c6dc9f4a2fdae31a49b93c800814f62 Nov 02 01:24:32.601689 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.601652 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk"] Nov 02 01:24:32.604656 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:32.604636 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ae4588_f166_4563_aae7_b34b5b053d0f.slice/crio-2c4476138a7bf2989aac4600b621afb29324711330f782eb69f9bbce13d98075 WatchSource:0}: Error finding container 2c4476138a7bf2989aac4600b621afb29324711330f782eb69f9bbce13d98075: Status 404 returned error can't find the container with id 2c4476138a7bf2989aac4600b621afb29324711330f782eb69f9bbce13d98075 Nov 02 01:24:32.651278 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.651253 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-58fr4"] Nov 02 01:24:32.654952 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:32.654931 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8cbfce_aaf5_4362_930e_b15539b2dbc4.slice/crio-e70bc567d899951d4b95692ad723aab0eee4d2d732c291c0b99e2ae4a2fd024a WatchSource:0}: Error finding container e70bc567d899951d4b95692ad723aab0eee4d2d732c291c0b99e2ae4a2fd024a: Status 404 returned error can't find the container with id e70bc567d899951d4b95692ad723aab0eee4d2d732c291c0b99e2ae4a2fd024a Nov 02 01:24:32.663312 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.663289 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xmlts"] Nov 02 01:24:32.665972 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:32.665939 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8618e1e8_4d71_4f7a_b2a1_1b950190f110.slice/crio-411ae169b7770debda9684191543d54f36415adc6398062ead31c1100c6c1d81 WatchSource:0}: Error finding container 411ae169b7770debda9684191543d54f36415adc6398062ead31c1100c6c1d81: Status 404 returned error can't find the container with id 411ae169b7770debda9684191543d54f36415adc6398062ead31c1100c6c1d81 Nov 02 01:24:32.724758 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.724731 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9sgpk"] Nov 02 01:24:32.728645 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.728628 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-frfx9"] Nov 02 01:24:32.738700 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:32.738658 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79ec42c_3696_40e7_94de_57aad9a9f92c.slice/crio-270cd97377d77588e4c335e61b9b1eba3f5ea162f96bc871d31f4f2e1213cc10 WatchSource:0}: Error finding container 270cd97377d77588e4c335e61b9b1eba3f5ea162f96bc871d31f4f2e1213cc10: Status 404 returned error can't find the container with id 270cd97377d77588e4c335e61b9b1eba3f5ea162f96bc871d31f4f2e1213cc10 Nov 02 01:24:32.738884 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:32.738866 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c285da3_2e11_498d_94f1_76c0ce604553.slice/crio-b136b6d11358e2172ebba343dc7590986e27110dc3800b9d61c9491d619191a5 WatchSource:0}: Error finding container b136b6d11358e2172ebba343dc7590986e27110dc3800b9d61c9491d619191a5: Status 404 returned error can't find the container with id b136b6d11358e2172ebba343dc7590986e27110dc3800b9d61c9491d619191a5 Nov 02 01:24:32.785362 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.785339 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zlq59"] Nov 02 01:24:32.787978 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:32.787953 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e031e5c_cb8a_42b9_b334_1cbe61a9c59f.slice/crio-06f2e60e44f6b1ae0d0ce0b7cdac42b43ef1a76ff92b2a849246f46e15896d45 WatchSource:0}: Error finding container 06f2e60e44f6b1ae0d0ce0b7cdac42b43ef1a76ff92b2a849246f46e15896d45: Status 404 returned error can't find the container with id 06f2e60e44f6b1ae0d0ce0b7cdac42b43ef1a76ff92b2a849246f46e15896d45 Nov 02 01:24:32.935710 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:32.935655 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:33.132985 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.132955 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:33.258856 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.258825 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:33.259011 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.258919 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:33.259011 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.258947 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:33.311396 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.311370 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 01:24:33.311750 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.311495 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-82k56\"" Nov 02 01:24:33.311750 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.311512 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6qqtd\"" Nov 02 01:24:33.311750 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.311515 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 01:24:33.311750 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.311515 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qtccq\"" Nov 02 01:24:33.474932 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.474853 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" event={"ID":"4abd5311-4a92-4ac5-8aa0-779dd8774577","Type":"ContainerStarted","Data":"780e6524fb0d75e94a6e9d5e7000be249c6dc9f4a2fdae31a49b93c800814f62"} Nov 02 01:24:33.476084 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.476059 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d9wwq" event={"ID":"363dffa0-70cf-4fe0-996f-3c4d2f3b1a50","Type":"ContainerStarted","Data":"e1fbaf835ff8f3a7565f0d4aefabd3fbdfac25aa4676e785fad3e60cc44e6a38"} Nov 02 01:24:33.477184 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.477165 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tgsq4" event={"ID":"91cb4fd5-ed27-4a79-b172-be80b203bf0f","Type":"ContainerStarted","Data":"d1b449089fc1b096507ccc81210b0ab22d0eef7f4816aa2fdafe9b3542bdbd4a"} Nov 02 01:24:33.478537 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.478512 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-v8sdv" event={"ID":"eb8ad2e6-7e4f-4c0a-9ea3-99ed98344ba6","Type":"ContainerStarted","Data":"023abcb077ee14f0fa843835312135bbbce16c6625c3302fac7f4b4b237f4e3c"} Nov 02 01:24:33.478629 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.478550 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-v8sdv" event={"ID":"eb8ad2e6-7e4f-4c0a-9ea3-99ed98344ba6","Type":"ContainerStarted","Data":"aef0f7510a57f5b33b937b4d573d2417f0dbf9f2f55ee57de1f762e2ea09b55d"} Nov 02 01:24:33.479610 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.479593 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9sgpk" event={"ID":"e79ec42c-3696-40e7-94de-57aad9a9f92c","Type":"ContainerStarted","Data":"270cd97377d77588e4c335e61b9b1eba3f5ea162f96bc871d31f4f2e1213cc10"} Nov 02 01:24:33.480614 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.480589 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" event={"ID":"7c285da3-2e11-498d-94f1-76c0ce604553","Type":"ContainerStarted","Data":"b136b6d11358e2172ebba343dc7590986e27110dc3800b9d61c9491d619191a5"} Nov 02 01:24:33.481518 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.481498 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" event={"ID":"35ae4588-f166-4563-aae7-b34b5b053d0f","Type":"ContainerStarted","Data":"2c4476138a7bf2989aac4600b621afb29324711330f782eb69f9bbce13d98075"} Nov 02 01:24:33.482749 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.482727 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xmlts" event={"ID":"8618e1e8-4d71-4f7a-b2a1-1b950190f110","Type":"ContainerStarted","Data":"b60bd0484095e7ef6f5378a8d5ff77dfc415f8bb7ca258f98e8f393d4177441b"} Nov 02 01:24:33.482831 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.482764 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xmlts" event={"ID":"8618e1e8-4d71-4f7a-b2a1-1b950190f110","Type":"ContainerStarted","Data":"411ae169b7770debda9684191543d54f36415adc6398062ead31c1100c6c1d81"} Nov 02 01:24:33.483951 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.483931 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zlq59" event={"ID":"2e031e5c-cb8a-42b9-b334-1cbe61a9c59f","Type":"ContainerStarted","Data":"d87991f394ee8ce20d5dba86804fc45ae69ae3de966a8e1bb176bc51e722b49e"} Nov 02 01:24:33.484018 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.483971 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zlq59" event={"ID":"2e031e5c-cb8a-42b9-b334-1cbe61a9c59f","Type":"ContainerStarted","Data":"06f2e60e44f6b1ae0d0ce0b7cdac42b43ef1a76ff92b2a849246f46e15896d45"} Nov 02 01:24:33.484972 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.484955 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-58fr4" event={"ID":"2c8cbfce-aaf5-4362-930e-b15539b2dbc4","Type":"ContainerStarted","Data":"e70bc567d899951d4b95692ad723aab0eee4d2d732c291c0b99e2ae4a2fd024a"} Nov 02 01:24:33.548084 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.548056 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:33.548084 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.548086 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-metrics-tls\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:33.548225 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.548130 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:33.548225 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.548156 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6sdmz\" (UID: \"59ccadf3-202d-481d-8c8b-c4fa86c92b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:33.548225 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.548186 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:33.548225 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:33.548203 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 01:24:33.548225 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.548214 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:33.548225 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:33.548223 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5954c6588f-7h92t: secret "image-registry-tls" not found Nov 02 01:24:33.548489 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.548254 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ntpqz\" (UID: \"fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:33.548489 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:33.548258 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 01:24:33.548489 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:33.548273 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls podName:7c2b846b-5c8b-4982-abc9-809fdbffe82d nodeName:}" failed. No retries permitted until 2026-11-02 01:24:35.548260124 +0000 UTC m=+35.710669045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls") pod "image-registry-5954c6588f-7h92t" (UID: "7c2b846b-5c8b-4982-abc9-809fdbffe82d") : secret "image-registry-tls" not found Nov 02 01:24:33.548489 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:33.548206 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:24:33.548489 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:33.548298 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 01:24:33.548489 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:33.548291 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle podName:bdebb5b6-76a2-48c8-810d-6f80549a4f3c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:35.548285193 +0000 UTC m=+35.710694113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle") pod "router-default-6cb67dcd4c-zk2vs" (UID: "bdebb5b6-76a2-48c8-810d-6f80549a4f3c") : configmap references non-existent config key: service-ca.crt Nov 02 01:24:33.548489 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:33.548317 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 01:24:33.548489 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:33.548336 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69469b6ffb-qmvll: secret "image-registry-tls" not found Nov 02 01:24:33.548489 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:33.548341 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-cluster-monitoring-operator-tls podName:59ccadf3-202d-481d-8c8b-c4fa86c92b8b nodeName:}" failed. No retries permitted until 2026-11-02 01:24:35.548324979 +0000 UTC m=+35.710733899 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6sdmz" (UID: "59ccadf3-202d-481d-8c8b-c4fa86c92b8b") : secret "cluster-monitoring-operator-tls" not found Nov 02 01:24:33.548489 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:33.548361 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-metrics-tls podName:da5a0b9c-57f5-4741-94f3-fb9fe7b7e699 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:35.5483512 +0000 UTC m=+35.710760120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-metrics-tls") pod "dns-default-qmrv6" (UID: "da5a0b9c-57f5-4741-94f3-fb9fe7b7e699") : secret "dns-default-metrics-tls" not found Nov 02 01:24:33.548489 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:33.548362 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 01:24:33.548489 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:33.548381 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls podName:d5c07387-ae45-48d3-8b9d-8f328bdfd63c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:35.548371402 +0000 UTC m=+35.710780325 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls") pod "image-registry-69469b6ffb-qmvll" (UID: "d5c07387-ae45-48d3-8b9d-8f328bdfd63c") : secret "image-registry-tls" not found Nov 02 01:24:33.548489 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:33.548408 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs podName:bdebb5b6-76a2-48c8-810d-6f80549a4f3c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:35.548396702 +0000 UTC m=+35.710805622 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs") pod "router-default-6cb67dcd4c-zk2vs" (UID: "bdebb5b6-76a2-48c8-810d-6f80549a4f3c") : secret "router-metrics-certs-default" not found Nov 02 01:24:33.548489 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:33.548431 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert podName:fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:35.548423883 +0000 UTC m=+35.710832806 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ntpqz" (UID: "fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41") : secret "networking-console-plugin-cert" not found Nov 02 01:24:33.548977 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:33.548482 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:33.548977 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:33.548617 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 01:24:33.548977 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:33.548686 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls podName:b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b nodeName:}" failed. No retries permitted until 2026-11-02 01:24:35.548651082 +0000 UTC m=+35.711060016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-brlzq" (UID: "b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b") : secret "samples-operator-tls" not found Nov 02 01:24:34.031064 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:34.029731 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-v8sdv" podStartSLOduration=391.029710095 podStartE2EDuration="6m31.029710095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:33.914203404 +0000 UTC m=+34.076612348" watchObservedRunningTime="2026-11-02 01:24:34.029710095 +0000 UTC m=+34.192119037" Nov 02 01:24:34.055401 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:34.054528 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs\") pod \"network-metrics-daemon-flwvx\" (UID: \"d628518e-2ab7-4acb-acf7-6fc96f359a64\") " pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:34.055401 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:34.054648 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:34.055401 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:34.055302 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 01:24:34.055401 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:34.055367 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs podName:d628518e-2ab7-4acb-acf7-6fc96f359a64 nodeName:}" failed. No retries permitted until 2026-11-02 01:25:06.055347293 +0000 UTC m=+66.217756227 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs") pod "network-metrics-daemon-flwvx" (UID: "d628518e-2ab7-4acb-acf7-6fc96f359a64") : secret "metrics-daemon-secret" not found Nov 02 01:24:34.066730 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:34.064058 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:34.146970 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:34.146882 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xmlts" podStartSLOduration=391.146860496 podStartE2EDuration="6m31.146860496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:34.030945795 +0000 UTC m=+34.193354738" watchObservedRunningTime="2026-11-02 01:24:34.146860496 +0000 UTC m=+34.309269441" Nov 02 01:24:34.156768 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:34.155957 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dzj7\" (UniqueName: \"kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7\") pod \"network-check-target-kgbsx\" (UID: \"85784876-9397-4bec-8ade-878a15c09f40\") " pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:34.159639 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:34.159589 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dzj7\" (UniqueName: \"kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7\") pod \"network-check-target-kgbsx\" (UID: \"85784876-9397-4bec-8ade-878a15c09f40\") " pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:34.189523 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:34.189323 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:34.240427 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:34.240401 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:34.625086 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:34.624978 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-zlq59" podStartSLOduration=391.624956948 podStartE2EDuration="6m31.624956948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:34.148502328 +0000 UTC m=+34.310911270" watchObservedRunningTime="2026-11-02 01:24:34.624956948 +0000 UTC m=+34.787365886" Nov 02 01:24:34.626646 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:34.626620 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7szx6"] Nov 02 01:24:34.631661 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:34.631633 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f97db3_814b_4849_ab8d_305dbcd3449c.slice/crio-4e417561d9def4122cb070f5244c8735d39aa9d61f3ac1d114ff9a4e5e5c4a4f WatchSource:0}: Error finding container 4e417561d9def4122cb070f5244c8735d39aa9d61f3ac1d114ff9a4e5e5c4a4f: Status 404 returned error can't find the container with id 4e417561d9def4122cb070f5244c8735d39aa9d61f3ac1d114ff9a4e5e5c4a4f Nov 02 01:24:34.696897 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:34.696861 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kgbsx"] Nov 02 01:24:34.707828 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:34.707802 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85784876_9397_4bec_8ade_878a15c09f40.slice/crio-ffc3d4e4c98fde42495e9d0bf5b1cc1a3a0e1b490ddd8ab72f086b1dd7fe9787 WatchSource:0}: Error finding container ffc3d4e4c98fde42495e9d0bf5b1cc1a3a0e1b490ddd8ab72f086b1dd7fe9787: Status 404 returned error can't find the container with id ffc3d4e4c98fde42495e9d0bf5b1cc1a3a0e1b490ddd8ab72f086b1dd7fe9787 Nov 02 01:24:35.497940 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:35.497892 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kgbsx" event={"ID":"85784876-9397-4bec-8ade-878a15c09f40","Type":"ContainerStarted","Data":"ffc3d4e4c98fde42495e9d0bf5b1cc1a3a0e1b490ddd8ab72f086b1dd7fe9787"} Nov 02 01:24:35.499744 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:35.499706 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7szx6" event={"ID":"79f97db3-814b-4849-ab8d-305dbcd3449c","Type":"ContainerStarted","Data":"4e417561d9def4122cb070f5244c8735d39aa9d61f3ac1d114ff9a4e5e5c4a4f"} Nov 02 01:24:35.570737 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:35.570691 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:35.570886 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:35.570739 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-metrics-tls\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:35.570886 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:35.570781 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:35.570886 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:35.570818 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6sdmz\" (UID: \"59ccadf3-202d-481d-8c8b-c4fa86c92b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:35.570886 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:35.570850 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:35.570886 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:35.570884 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:35.571290 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:35.570929 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ntpqz\" (UID: \"fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:35.571290 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:35.570978 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:35.571290 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:35.571171 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 01:24:35.571290 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:35.571228 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls podName:b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b nodeName:}" failed. No retries permitted until 2026-11-02 01:24:39.571210427 +0000 UTC m=+39.733619361 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-brlzq" (UID: "b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b") : secret "samples-operator-tls" not found Nov 02 01:24:35.571481 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:35.571302 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 01:24:35.571481 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:35.571317 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5954c6588f-7h92t: secret "image-registry-tls" not found Nov 02 01:24:35.571481 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:35.571353 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls podName:7c2b846b-5c8b-4982-abc9-809fdbffe82d nodeName:}" failed. No retries permitted until 2026-11-02 01:24:39.571340787 +0000 UTC m=+39.733749722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls") pod "image-registry-5954c6588f-7h92t" (UID: "7c2b846b-5c8b-4982-abc9-809fdbffe82d") : secret "image-registry-tls" not found Nov 02 01:24:35.571481 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:35.571412 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:24:35.571481 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:35.571439 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-metrics-tls podName:da5a0b9c-57f5-4741-94f3-fb9fe7b7e699 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:39.571429716 +0000 UTC m=+39.733838650 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-metrics-tls") pod "dns-default-qmrv6" (UID: "da5a0b9c-57f5-4741-94f3-fb9fe7b7e699") : secret "dns-default-metrics-tls" not found Nov 02 01:24:35.571719 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:35.571506 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle podName:bdebb5b6-76a2-48c8-810d-6f80549a4f3c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:39.571496727 +0000 UTC m=+39.733905646 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle") pod "router-default-6cb67dcd4c-zk2vs" (UID: "bdebb5b6-76a2-48c8-810d-6f80549a4f3c") : configmap references non-existent config key: service-ca.crt Nov 02 01:24:35.571719 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:35.571561 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 01:24:35.571719 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:35.571593 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-cluster-monitoring-operator-tls podName:59ccadf3-202d-481d-8c8b-c4fa86c92b8b nodeName:}" failed. No retries permitted until 2026-11-02 01:24:39.571583236 +0000 UTC m=+39.733992160 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6sdmz" (UID: "59ccadf3-202d-481d-8c8b-c4fa86c92b8b") : secret "cluster-monitoring-operator-tls" not found Nov 02 01:24:35.571719 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:35.571650 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 01:24:35.571719 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:35.571661 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69469b6ffb-qmvll: secret "image-registry-tls" not found Nov 02 01:24:35.571719 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:35.571712 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls podName:d5c07387-ae45-48d3-8b9d-8f328bdfd63c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:39.571700823 +0000 UTC m=+39.734109750 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls") pod "image-registry-69469b6ffb-qmvll" (UID: "d5c07387-ae45-48d3-8b9d-8f328bdfd63c") : secret "image-registry-tls" not found Nov 02 01:24:35.572045 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:35.571774 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 01:24:35.572045 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:35.571810 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs podName:bdebb5b6-76a2-48c8-810d-6f80549a4f3c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:39.571797892 +0000 UTC m=+39.734206826 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs") pod "router-default-6cb67dcd4c-zk2vs" (UID: "bdebb5b6-76a2-48c8-810d-6f80549a4f3c") : secret "router-metrics-certs-default" not found Nov 02 01:24:35.572045 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:35.571867 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 01:24:35.572045 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:35.571913 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert podName:fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:39.571903056 +0000 UTC m=+39.734311976 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ntpqz" (UID: "fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41") : secret "networking-console-plugin-cert" not found Nov 02 01:24:35.680081 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:35.680012 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:36.507445 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:36.506606 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d9wwq" event={"ID":"363dffa0-70cf-4fe0-996f-3c4d2f3b1a50","Type":"ContainerStarted","Data":"be62674601c3fbecce63e22fcc9f6276081d739b07adfa289edf629ca742e31a"} Nov 02 01:24:36.737699 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:36.737623 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:36.884889 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:36.884838 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:38.517603 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:38.516645 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/0.log" Nov 02 01:24:38.517603 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:38.516704 2567 generic.go:356] "Generic (PLEG): container finished" podID="7c285da3-2e11-498d-94f1-76c0ce604553" containerID="a188c7040a1bce29fe09de9fd39ecf989c99d451d972c6180b02c8f8d67463e3" exitCode=255 Nov 02 01:24:38.517603 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:38.516794 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" event={"ID":"7c285da3-2e11-498d-94f1-76c0ce604553","Type":"ContainerDied","Data":"a188c7040a1bce29fe09de9fd39ecf989c99d451d972c6180b02c8f8d67463e3"} Nov 02 01:24:38.517603 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:38.517109 2567 scope.go:122] "RemoveContainer" containerID="a188c7040a1bce29fe09de9fd39ecf989c99d451d972c6180b02c8f8d67463e3" Nov 02 01:24:38.522195 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:38.521661 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" event={"ID":"4abd5311-4a92-4ac5-8aa0-779dd8774577","Type":"ContainerStarted","Data":"1040377acf4b69c4a29c3ce7a8e07950d47512ae78348025b1a64d20c84493a8"} Nov 02 01:24:38.525830 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:38.525426 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tgsq4" event={"ID":"91cb4fd5-ed27-4a79-b172-be80b203bf0f","Type":"ContainerStarted","Data":"b2c932d315e10d88a096b2910463f995c257b4320f82b026d02975419da0db8d"} Nov 02 01:24:38.640106 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:38.638068 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d9wwq" podStartSLOduration=366.823506563 podStartE2EDuration="6m9.638048681s" totalImagesPullingTime="2.814542118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:36.624859284 +0000 UTC m=+36.787268229" watchObservedRunningTime="2026-11-02 01:24:38.638048681 +0000 UTC m=+38.800457625" Nov 02 01:24:38.756553 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:38.754890 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-tgsq4" podStartSLOduration=364.558339242 podStartE2EDuration="6m9.754874751s" totalImagesPullingTime="5.196535509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:38.754235299 +0000 UTC m=+38.916644253" watchObservedRunningTime="2026-11-02 01:24:38.754874751 +0000 UTC m=+38.917283692" Nov 02 01:24:38.872643 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:38.872278 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" podStartSLOduration=364.295279455 podStartE2EDuration="6m9.872257026s" totalImagesPullingTime="5.576977571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:38.870629755 +0000 UTC m=+39.033038698" watchObservedRunningTime="2026-11-02 01:24:38.872257026 +0000 UTC m=+39.034665973" Nov 02 01:24:39.308650 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.308613 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r2hxq"] Nov 02 01:24:39.337388 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.337357 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:39.357710 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.357679 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r2hxq"] Nov 02 01:24:39.390774 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.390738 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l67lg\"" Nov 02 01:24:39.390915 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.390847 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 01:24:39.418326 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.418295 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:39.418461 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.418341 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54db4b99-9007-429a-a267-b4924a8729e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:39.418521 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.418470 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57blk\" (UniqueName: \"kubernetes.io/projected/54db4b99-9007-429a-a267-b4924a8729e6-kube-api-access-57blk\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:39.418570 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.418519 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/54db4b99-9007-429a-a267-b4924a8729e6-data-volume\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:39.418638 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.418621 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54db4b99-9007-429a-a267-b4924a8729e6-crio-socket\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:39.449770 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.449744 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 01:24:39.519469 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.519428 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/54db4b99-9007-429a-a267-b4924a8729e6-data-volume\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:39.519962 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.519544 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54db4b99-9007-429a-a267-b4924a8729e6-crio-socket\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:39.519962 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.519641 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:39.519962 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.519696 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54db4b99-9007-429a-a267-b4924a8729e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:39.519962 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.519751 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57blk\" (UniqueName: \"kubernetes.io/projected/54db4b99-9007-429a-a267-b4924a8729e6-kube-api-access-57blk\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:39.519962 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.519808 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54db4b99-9007-429a-a267-b4924a8729e6-crio-socket\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:39.519962 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.519823 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/54db4b99-9007-429a-a267-b4924a8729e6-data-volume\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:39.519962 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.519898 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 01:24:39.519962 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.519922 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:24:39.519962 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.519961 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/54db4b99-9007-429a-a267-b4924a8729e6-kube-rbac-proxy-cm podName:54db4b99-9007-429a-a267-b4924a8729e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:40.019940853 +0000 UTC m=+40.182349774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/54db4b99-9007-429a-a267-b4924a8729e6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r2hxq" (UID: "54db4b99-9007-429a-a267-b4924a8729e6") : configmap "kube-rbac-proxy" not found Nov 02 01:24:39.520413 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.519979 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls podName:54db4b99-9007-429a-a267-b4924a8729e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:40.019969906 +0000 UTC m=+40.182378832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2hxq" (UID: "54db4b99-9007-429a-a267-b4924a8729e6") : secret "insights-runtime-extractor-tls" not found Nov 02 01:24:39.533416 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.533390 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/1.log" Nov 02 01:24:39.533812 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.533788 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/0.log" Nov 02 01:24:39.533922 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.533833 2567 generic.go:356] "Generic (PLEG): container finished" podID="7c285da3-2e11-498d-94f1-76c0ce604553" containerID="78b5c1012635d39793a7b978cf50e9388d5edea5afd509bfce4fa68c48bf2e5a" exitCode=255 Nov 02 01:24:39.534825 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.534230 2567 scope.go:122] "RemoveContainer" containerID="78b5c1012635d39793a7b978cf50e9388d5edea5afd509bfce4fa68c48bf2e5a" Nov 02 01:24:39.534825 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.534441 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-frfx9_openshift-console-operator(7c285da3-2e11-498d-94f1-76c0ce604553)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" podUID="7c285da3-2e11-498d-94f1-76c0ce604553" Nov 02 01:24:39.534825 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.534491 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" event={"ID":"7c285da3-2e11-498d-94f1-76c0ce604553","Type":"ContainerDied","Data":"78b5c1012635d39793a7b978cf50e9388d5edea5afd509bfce4fa68c48bf2e5a"} Nov 02 01:24:39.534825 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.534542 2567 scope.go:122] "RemoveContainer" containerID="a188c7040a1bce29fe09de9fd39ecf989c99d451d972c6180b02c8f8d67463e3" Nov 02 01:24:39.535679 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.535646 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:39.539142 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.538754 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" event={"ID":"35ae4588-f166-4563-aae7-b34b5b053d0f","Type":"ContainerStarted","Data":"9413267b71c2d2bc04d27f9909f37ce17cf05ff74f22be400d8e43e5303adec8"} Nov 02 01:24:39.551001 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.549866 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-5bwtl-qgqdr"] Nov 02 01:24:39.574595 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.573339 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:39.584335 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.584280 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57blk\" (UniqueName: \"kubernetes.io/projected/54db4b99-9007-429a-a267-b4924a8729e6-kube-api-access-57blk\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:39.611370 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.611344 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-5bwtl-qgqdr"] Nov 02 01:24:39.621109 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.621084 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:39.621211 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.621144 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:39.621253 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.621211 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ntpqz\" (UID: \"fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:39.621306 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.621290 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:39.622770 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.622661 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 01:24:39.622770 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.622697 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69469b6ffb-qmvll: secret "image-registry-tls" not found Nov 02 01:24:39.622973 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.622953 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 01:24:39.623063 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.623013 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs podName:bdebb5b6-76a2-48c8-810d-6f80549a4f3c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:47.622992745 +0000 UTC m=+47.785401669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs") pod "router-default-6cb67dcd4c-zk2vs" (UID: "bdebb5b6-76a2-48c8-810d-6f80549a4f3c") : secret "router-metrics-certs-default" not found Nov 02 01:24:39.623143 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.623065 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 01:24:39.623143 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.623102 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert podName:fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:47.623090869 +0000 UTC m=+47.785499802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ntpqz" (UID: "fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41") : secret "networking-console-plugin-cert" not found Nov 02 01:24:39.624102 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.623319 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls podName:d5c07387-ae45-48d3-8b9d-8f328bdfd63c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:47.62329369 +0000 UTC m=+47.785702633 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls") pod "image-registry-69469b6ffb-qmvll" (UID: "d5c07387-ae45-48d3-8b9d-8f328bdfd63c") : secret "image-registry-tls" not found Nov 02 01:24:39.624102 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.623387 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 01:24:39.624102 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.623497 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls podName:b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b nodeName:}" failed. No retries permitted until 2026-11-02 01:24:47.623487578 +0000 UTC m=+47.785896527 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-brlzq" (UID: "b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b") : secret "samples-operator-tls" not found Nov 02 01:24:39.624102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.623616 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:39.624102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.623663 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-metrics-tls\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:39.624102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.623709 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:39.624102 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.623720 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 01:24:39.624102 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.623732 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5954c6588f-7h92t: secret "image-registry-tls" not found Nov 02 01:24:39.624102 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.623742 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6sdmz\" (UID: \"59ccadf3-202d-481d-8c8b-c4fa86c92b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:39.624102 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.623767 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls podName:7c2b846b-5c8b-4982-abc9-809fdbffe82d nodeName:}" failed. No retries permitted until 2026-11-02 01:24:47.623755132 +0000 UTC m=+47.786164067 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls") pod "image-registry-5954c6588f-7h92t" (UID: "7c2b846b-5c8b-4982-abc9-809fdbffe82d") : secret "image-registry-tls" not found Nov 02 01:24:39.624102 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.623820 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 01:24:39.624102 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.623841 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 01:24:39.624102 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.623849 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-metrics-tls podName:da5a0b9c-57f5-4741-94f3-fb9fe7b7e699 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:47.623837101 +0000 UTC m=+47.786246024 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-metrics-tls") pod "dns-default-qmrv6" (UID: "da5a0b9c-57f5-4741-94f3-fb9fe7b7e699") : secret "dns-default-metrics-tls" not found Nov 02 01:24:39.624102 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.623883 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-cluster-monitoring-operator-tls podName:59ccadf3-202d-481d-8c8b-c4fa86c92b8b nodeName:}" failed. No retries permitted until 2026-11-02 01:24:47.62386848 +0000 UTC m=+47.786277434 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6sdmz" (UID: "59ccadf3-202d-481d-8c8b-c4fa86c92b8b") : secret "cluster-monitoring-operator-tls" not found Nov 02 01:24:39.624102 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:39.623949 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle podName:bdebb5b6-76a2-48c8-810d-6f80549a4f3c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:47.623939117 +0000 UTC m=+47.786348040 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle") pod "router-default-6cb67dcd4c-zk2vs" (UID: "bdebb5b6-76a2-48c8-810d-6f80549a4f3c") : configmap references non-existent config key: service-ca.crt Nov 02 01:24:39.673067 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.673030 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pt8bk\"" Nov 02 01:24:39.691480 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.691447 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:39.724950 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.724911 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-trusted-ca-bundle\") pod \"periodic-gathering-5bwtl-qgqdr\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:39.725151 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.725036 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drbjx\" (UniqueName: \"kubernetes.io/projected/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-kube-api-access-drbjx\") pod \"periodic-gathering-5bwtl-qgqdr\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:39.725151 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.725090 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-archives-path\") pod \"periodic-gathering-5bwtl-qgqdr\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:39.725428 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.725205 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-service-ca-bundle\") pod \"periodic-gathering-5bwtl-qgqdr\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:39.826685 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.826591 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-service-ca-bundle\") pod \"periodic-gathering-5bwtl-qgqdr\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:39.826685 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.826660 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-trusted-ca-bundle\") pod \"periodic-gathering-5bwtl-qgqdr\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:39.826918 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.826776 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drbjx\" (UniqueName: \"kubernetes.io/projected/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-kube-api-access-drbjx\") pod \"periodic-gathering-5bwtl-qgqdr\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:39.826918 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.826830 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-archives-path\") pod \"periodic-gathering-5bwtl-qgqdr\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:39.827281 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.827253 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-archives-path\") pod \"periodic-gathering-5bwtl-qgqdr\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:39.827749 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.827728 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-service-ca-bundle\") pod \"periodic-gathering-5bwtl-qgqdr\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:39.828448 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.828424 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-trusted-ca-bundle\") pod \"periodic-gathering-5bwtl-qgqdr\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:39.848162 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.848128 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:39.888865 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.888819 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" podStartSLOduration=364.066510777 podStartE2EDuration="6m9.888805049s" totalImagesPullingTime="5.822294272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:39.887344431 +0000 UTC m=+40.049753366" watchObservedRunningTime="2026-11-02 01:24:39.888805049 +0000 UTC m=+40.051213990" Nov 02 01:24:39.895218 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.895192 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drbjx\" (UniqueName: \"kubernetes.io/projected/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-kube-api-access-drbjx\") pod \"periodic-gathering-5bwtl-qgqdr\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:39.916129 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.915936 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:39.995275 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:39.995241 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:40.028524 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.028493 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:40.028686 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.028546 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54db4b99-9007-429a-a267-b4924a8729e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:40.028686 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:40.028645 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:24:40.028816 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:40.028695 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 01:24:40.028816 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:40.028726 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls podName:54db4b99-9007-429a-a267-b4924a8729e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:41.028706519 +0000 UTC m=+41.191115450 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2hxq" (UID: "54db4b99-9007-429a-a267-b4924a8729e6") : secret "insights-runtime-extractor-tls" not found Nov 02 01:24:40.028816 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:40.028743 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/54db4b99-9007-429a-a267-b4924a8729e6-kube-rbac-proxy-cm podName:54db4b99-9007-429a-a267-b4924a8729e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:41.028734249 +0000 UTC m=+41.191143174 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/54db4b99-9007-429a-a267-b4924a8729e6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r2hxq" (UID: "54db4b99-9007-429a-a267-b4924a8729e6") : configmap "kube-rbac-proxy" not found Nov 02 01:24:40.093454 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.093380 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-q29rd"] Nov 02 01:24:40.110221 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.110191 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n5ppw"] Nov 02 01:24:40.110938 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.110461 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q29rd" Nov 02 01:24:40.132458 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.132209 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n5ppw" Nov 02 01:24:40.146510 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.146150 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:40.155749 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.155724 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-q29rd"] Nov 02 01:24:40.163914 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.162985 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 01:24:40.163914 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.162998 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-klgj9\"" Nov 02 01:24:40.163914 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.163057 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 01:24:40.168571 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.168543 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n5ppw"] Nov 02 01:24:40.230722 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.230692 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgsc7\" (UniqueName: \"kubernetes.io/projected/e6c17513-0478-4a05-94dd-0cdad5facf06-kube-api-access-rgsc7\") pod \"migrator-66df5d496b-q29rd\" (UID: \"e6c17513-0478-4a05-94dd-0cdad5facf06\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q29rd" Nov 02 01:24:40.230909 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.230887 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc4n8\" (UniqueName: \"kubernetes.io/projected/476d365d-dd0a-41e6-a186-5e92551be763-kube-api-access-dc4n8\") pod \"migrator-66df5d496b-n5ppw\" (UID: \"476d365d-dd0a-41e6-a186-5e92551be763\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n5ppw" Nov 02 01:24:40.298247 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.298204 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:40.332382 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.332347 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc4n8\" (UniqueName: \"kubernetes.io/projected/476d365d-dd0a-41e6-a186-5e92551be763-kube-api-access-dc4n8\") pod \"migrator-66df5d496b-n5ppw\" (UID: \"476d365d-dd0a-41e6-a186-5e92551be763\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n5ppw" Nov 02 01:24:40.332568 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.332490 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgsc7\" (UniqueName: \"kubernetes.io/projected/e6c17513-0478-4a05-94dd-0cdad5facf06-kube-api-access-rgsc7\") pod \"migrator-66df5d496b-q29rd\" (UID: \"e6c17513-0478-4a05-94dd-0cdad5facf06\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q29rd" Nov 02 01:24:40.395389 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.395329 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgsc7\" (UniqueName: \"kubernetes.io/projected/e6c17513-0478-4a05-94dd-0cdad5facf06-kube-api-access-rgsc7\") pod \"migrator-66df5d496b-q29rd\" (UID: \"e6c17513-0478-4a05-94dd-0cdad5facf06\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q29rd" Nov 02 01:24:40.395660 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.395634 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc4n8\" (UniqueName: \"kubernetes.io/projected/476d365d-dd0a-41e6-a186-5e92551be763-kube-api-access-dc4n8\") pod \"migrator-66df5d496b-n5ppw\" (UID: \"476d365d-dd0a-41e6-a186-5e92551be763\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n5ppw" Nov 02 01:24:40.450220 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.450191 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:40.451206 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.451183 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q29rd" Nov 02 01:24:40.462354 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.462331 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n5ppw" Nov 02 01:24:40.472210 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.472187 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-5bwtl-qgqdr"] Nov 02 01:24:40.547887 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.547037 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9sgpk" event={"ID":"e79ec42c-3696-40e7-94de-57aad9a9f92c","Type":"ContainerStarted","Data":"0b4cad791985979ffb071783d0899c5e082c043b709611a7057c5677b1bfa291"} Nov 02 01:24:40.561618 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.560786 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/1.log" Nov 02 01:24:40.561618 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.561273 2567 scope.go:122] "RemoveContainer" containerID="78b5c1012635d39793a7b978cf50e9388d5edea5afd509bfce4fa68c48bf2e5a" Nov 02 01:24:40.561618 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:40.561525 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-frfx9_openshift-console-operator(7c285da3-2e11-498d-94f1-76c0ce604553)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" podUID="7c285da3-2e11-498d-94f1-76c0ce604553" Nov 02 01:24:40.568059 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.567276 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-58fr4" event={"ID":"2c8cbfce-aaf5-4362-930e-b15539b2dbc4","Type":"ContainerStarted","Data":"6e8e1fc9bb270579f590f583f832995ec29ad10a101eb9fcff9cdb3ea10f01d6"} Nov 02 01:24:40.570046 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.569976 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" event={"ID":"eb932801-604f-430e-bfe3-8d4c9a1a7ca6","Type":"ContainerStarted","Data":"194ec6187c7b572797fd8f78e02cb66a027fb1b3f1f870aa4adda003f033d9e7"} Nov 02 01:24:40.664009 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.663918 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9sgpk" podStartSLOduration=2.297513808 podStartE2EDuration="9.663901055s" totalImagesPullingTime="7.366387247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:40.663713384 +0000 UTC m=+40.826122328" watchObservedRunningTime="2026-11-02 01:24:40.663901055 +0000 UTC m=+40.826309999" Nov 02 01:24:40.871438 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.871407 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-q29rd"] Nov 02 01:24:40.899931 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.899879 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-58fr4" podStartSLOduration=357.163868985 podStartE2EDuration="6m4.899864101s" totalImagesPullingTime="7.735995116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:40.898504732 +0000 UTC m=+41.060913677" watchObservedRunningTime="2026-11-02 01:24:40.899864101 +0000 UTC m=+41.062273044" Nov 02 01:24:40.903487 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.903452 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:40.904153 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:40.904125 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c17513_0478_4a05_94dd_0cdad5facf06.slice/crio-9fcaa49884d943ea43a86669a9512e72925c2d85280c94b971f08c50e553d79f WatchSource:0}: Error finding container 9fcaa49884d943ea43a86669a9512e72925c2d85280c94b971f08c50e553d79f: Status 404 returned error can't find the container with id 9fcaa49884d943ea43a86669a9512e72925c2d85280c94b971f08c50e553d79f Nov 02 01:24:40.912108 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:40.912085 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n5ppw"] Nov 02 01:24:40.971882 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:40.971855 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476d365d_dd0a_41e6_a186_5e92551be763.slice/crio-8d2fee6e42afcc68e2cbe0d970b893f2203dbf493985ccb97070845466de3e72 WatchSource:0}: Error finding container 8d2fee6e42afcc68e2cbe0d970b893f2203dbf493985ccb97070845466de3e72: Status 404 returned error can't find the container with id 8d2fee6e42afcc68e2cbe0d970b893f2203dbf493985ccb97070845466de3e72 Nov 02 01:24:41.041369 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.041339 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:41.041497 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.041393 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54db4b99-9007-429a-a267-b4924a8729e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:41.041497 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:41.041480 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 01:24:41.041593 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:41.041501 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:24:41.041593 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:41.041525 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/54db4b99-9007-429a-a267-b4924a8729e6-kube-rbac-proxy-cm podName:54db4b99-9007-429a-a267-b4924a8729e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:43.04151059 +0000 UTC m=+43.203919523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/54db4b99-9007-429a-a267-b4924a8729e6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r2hxq" (UID: "54db4b99-9007-429a-a267-b4924a8729e6") : configmap "kube-rbac-proxy" not found Nov 02 01:24:41.041593 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:41.041555 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls podName:54db4b99-9007-429a-a267-b4924a8729e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:43.041541588 +0000 UTC m=+43.203950513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2hxq" (UID: "54db4b99-9007-429a-a267-b4924a8729e6") : secret "insights-runtime-extractor-tls" not found Nov 02 01:24:41.050929 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.050900 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:41.130828 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.130799 2567 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-qkqj6"] Nov 02 01:24:41.154837 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.154117 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-qkqj6" Nov 02 01:24:41.192108 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.192068 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-qkqj6"] Nov 02 01:24:41.209653 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.209579 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 01:24:41.209816 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.209680 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 01:24:41.209816 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.209706 2567 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cpkxg\"" Nov 02 01:24:41.210110 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.210071 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 01:24:41.210933 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.210899 2567 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 01:24:41.244290 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.244258 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/028b56de-c4b9-47c1-8bfa-761877176429-signing-key\") pod \"service-ca-5fffc97bdf-qkqj6\" (UID: \"028b56de-c4b9-47c1-8bfa-761877176429\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qkqj6" Nov 02 01:24:41.244522 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.244487 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/028b56de-c4b9-47c1-8bfa-761877176429-signing-cabundle\") pod \"service-ca-5fffc97bdf-qkqj6\" (UID: \"028b56de-c4b9-47c1-8bfa-761877176429\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qkqj6" Nov 02 01:24:41.244658 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.244634 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4h6s\" (UniqueName: \"kubernetes.io/projected/028b56de-c4b9-47c1-8bfa-761877176429-kube-api-access-k4h6s\") pod \"service-ca-5fffc97bdf-qkqj6\" (UID: \"028b56de-c4b9-47c1-8bfa-761877176429\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qkqj6" Nov 02 01:24:41.346040 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.345996 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/028b56de-c4b9-47c1-8bfa-761877176429-signing-cabundle\") pod \"service-ca-5fffc97bdf-qkqj6\" (UID: \"028b56de-c4b9-47c1-8bfa-761877176429\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qkqj6" Nov 02 01:24:41.346190 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.346103 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4h6s\" (UniqueName: \"kubernetes.io/projected/028b56de-c4b9-47c1-8bfa-761877176429-kube-api-access-k4h6s\") pod \"service-ca-5fffc97bdf-qkqj6\" (UID: \"028b56de-c4b9-47c1-8bfa-761877176429\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qkqj6" Nov 02 01:24:41.346190 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.346165 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/028b56de-c4b9-47c1-8bfa-761877176429-signing-key\") pod \"service-ca-5fffc97bdf-qkqj6\" (UID: \"028b56de-c4b9-47c1-8bfa-761877176429\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qkqj6" Nov 02 01:24:41.346992 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.346968 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/028b56de-c4b9-47c1-8bfa-761877176429-signing-cabundle\") pod \"service-ca-5fffc97bdf-qkqj6\" (UID: \"028b56de-c4b9-47c1-8bfa-761877176429\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qkqj6" Nov 02 01:24:41.348770 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.348746 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/028b56de-c4b9-47c1-8bfa-761877176429-signing-key\") pod \"service-ca-5fffc97bdf-qkqj6\" (UID: \"028b56de-c4b9-47c1-8bfa-761877176429\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qkqj6" Nov 02 01:24:41.360283 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.360250 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:41.411168 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.411142 2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4h6s\" (UniqueName: \"kubernetes.io/projected/028b56de-c4b9-47c1-8bfa-761877176429-kube-api-access-k4h6s\") pod \"service-ca-5fffc97bdf-qkqj6\" (UID: \"028b56de-c4b9-47c1-8bfa-761877176429\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qkqj6" Nov 02 01:24:41.478977 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.478910 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-qkqj6" Nov 02 01:24:41.517011 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.516983 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:41.580374 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.580344 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kgbsx" event={"ID":"85784876-9397-4bec-8ade-878a15c09f40","Type":"ContainerStarted","Data":"9815f722d32de1f845946052057b8173894abac7c6b71fb69aabd8ec999c0b87"} Nov 02 01:24:41.580752 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.580455 2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:41.582346 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.582285 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" event={"ID":"eb932801-604f-430e-bfe3-8d4c9a1a7ca6","Type":"ContainerStarted","Data":"d561c8c28f147177b3d69d7c35d73e9821dadf4125eec9721a577287390da5a3"} Nov 02 01:24:41.583633 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.583606 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n5ppw" event={"ID":"476d365d-dd0a-41e6-a186-5e92551be763","Type":"ContainerStarted","Data":"8d2fee6e42afcc68e2cbe0d970b893f2203dbf493985ccb97070845466de3e72"} Nov 02 01:24:41.584823 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.584796 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q29rd" event={"ID":"e6c17513-0478-4a05-94dd-0cdad5facf06","Type":"ContainerStarted","Data":"9fcaa49884d943ea43a86669a9512e72925c2d85280c94b971f08c50e553d79f"} Nov 02 01:24:41.815569 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.814113 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" podStartSLOduration=2.814096307 podStartE2EDuration="2.814096307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:41.81390685 +0000 UTC m=+41.976315795" watchObservedRunningTime="2026-11-02 01:24:41.814096307 +0000 UTC m=+41.976505251" Nov 02 01:24:41.815569 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.815368 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kgbsx" podStartSLOduration=34.520695506 podStartE2EDuration="40.815355221s" totalImagesPullingTime="6.294659715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:41.696565211 +0000 UTC m=+41.858974151" watchObservedRunningTime="2026-11-02 01:24:41.815355221 +0000 UTC m=+41.977764162" Nov 02 01:24:41.836771 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.836733 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:41.871906 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:41.871877 2567 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-qkqj6"] Nov 02 01:24:41.912438 ip-10-0-129-6 kubenswrapper[2567]: W1102 01:24:41.912404 2567 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028b56de_c4b9_47c1_8bfa_761877176429.slice/crio-0f2e12d4751d7b88c57875fc67990526ffd54c6b50370cda54f8241789bdf974 WatchSource:0}: Error finding container 0f2e12d4751d7b88c57875fc67990526ffd54c6b50370cda54f8241789bdf974: Status 404 returned error can't find the container with id 0f2e12d4751d7b88c57875fc67990526ffd54c6b50370cda54f8241789bdf974 Nov 02 01:24:42.035991 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:42.035963 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:42.273347 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:42.273318 2567 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:42.273517 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:42.273376 2567 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:42.273791 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:42.273775 2567 scope.go:122] "RemoveContainer" containerID="78b5c1012635d39793a7b978cf50e9388d5edea5afd509bfce4fa68c48bf2e5a" Nov 02 01:24:42.274011 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:42.273988 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-frfx9_openshift-console-operator(7c285da3-2e11-498d-94f1-76c0ce604553)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" podUID="7c285da3-2e11-498d-94f1-76c0ce604553" Nov 02 01:24:42.591051 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:42.590398 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-qkqj6" event={"ID":"028b56de-c4b9-47c1-8bfa-761877176429","Type":"ContainerStarted","Data":"16b991f977e5d23248f1decc7d00ad3b35fc275711b26a74ff8075be89dfa1ae"} Nov 02 01:24:42.591051 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:42.590478 2567 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-qkqj6" event={"ID":"028b56de-c4b9-47c1-8bfa-761877176429","Type":"ContainerStarted","Data":"0f2e12d4751d7b88c57875fc67990526ffd54c6b50370cda54f8241789bdf974"} Nov 02 01:24:42.707628 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:42.707577 2567 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-qkqj6" podStartSLOduration=1.707563749 podStartE2EDuration="1.707563749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:24:42.706743095 +0000 UTC m=+42.869152039" watchObservedRunningTime="2026-11-02 01:24:42.707563749 +0000 UTC m=+42.869972690" Nov 02 01:24:43.066923 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:43.066898 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:43.066999 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:43.066936 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54db4b99-9007-429a-a267-b4924a8729e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:43.067052 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:43.067041 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 01:24:43.067090 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:43.067058 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:24:43.067090 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:43.067088 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/54db4b99-9007-429a-a267-b4924a8729e6-kube-rbac-proxy-cm podName:54db4b99-9007-429a-a267-b4924a8729e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:47.067075957 +0000 UTC m=+47.229484877 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/54db4b99-9007-429a-a267-b4924a8729e6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r2hxq" (UID: "54db4b99-9007-429a-a267-b4924a8729e6") : configmap "kube-rbac-proxy" not found Nov 02 01:24:43.067193 ip-10-0-129-6 kubenswrapper[2567]: E1102 01:24:43.067113 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls podName:54db4b99-9007-429a-a267-b4924a8729e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:47.067100069 +0000 UTC m=+47.229509016 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2hxq" (UID: "54db4b99-9007-429a-a267-b4924a8729e6") : secret "insights-runtime-extractor-tls" not found Nov 02 01:24:43.249736 ip-10-0-129-6 kubenswrapper[2567]: I1102 01:24:43.249104 2567 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 01:24:43.249297 ip-10-0-129-6 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 01:24:43.275049 ip-10-0-129-6 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 01:24:43.275213 ip-10-0-129-6 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 01:24:43.275260 ip-10-0-129-6 systemd[1]: kubelet.service: Consumed 3.105s CPU time. Nov 02 01:24:43.286259 ip-10-0-129-6 systemd[1]: Starting Kubernetes Kubelet... Nov 02 01:24:43.357774 ip-10-0-129-6 kubenswrapper[5044]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 01:24:43.357774 ip-10-0-129-6 kubenswrapper[5044]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 01:24:43.357774 ip-10-0-129-6 kubenswrapper[5044]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 01:24:43.357774 ip-10-0-129-6 kubenswrapper[5044]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 01:24:43.359759 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359688 5044 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:24:43.359759 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359757 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:24:43.359759 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359761 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:24:43.359759 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359764 5044 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:24:43.359882 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359767 5044 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:24:43.359882 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359771 5044 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:24:43.359882 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359773 5044 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:24:43.359882 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359776 5044 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:24:43.359882 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359779 5044 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:24:43.359882 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359781 5044 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:24:43.359882 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359784 5044 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:24:43.359882 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359787 5044 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:24:43.359882 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359791 5044 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:24:43.359882 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359796 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:24:43.359882 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359799 5044 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:24:43.359882 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359802 5044 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:24:43.359882 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359805 5044 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:24:43.359882 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359808 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:24:43.359882 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359811 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:24:43.359882 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359813 5044 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:24:43.359882 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359816 5044 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:24:43.359882 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359819 5044 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:24:43.359882 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359822 5044 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:24:43.359882 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359825 5044 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:24:43.360372 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359827 5044 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:24:43.360372 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359830 5044 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:24:43.360372 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359833 5044 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:24:43.360372 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359836 5044 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:24:43.360372 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359838 5044 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:24:43.360372 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359841 5044 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:24:43.360372 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359843 5044 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:24:43.360372 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359846 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:24:43.360372 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359848 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:24:43.360372 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359851 5044 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:24:43.360372 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359854 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:24:43.360372 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359857 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:24:43.360372 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359860 5044 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:24:43.360372 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359862 5044 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:24:43.360372 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359865 5044 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:24:43.360372 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359868 5044 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:24:43.360372 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359871 5044 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:24:43.360372 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359874 5044 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:24:43.360372 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359876 5044 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:24:43.360372 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359879 5044 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:24:43.360912 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359888 5044 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:24:43.360912 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359891 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:24:43.360912 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359894 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:24:43.360912 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359897 5044 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:24:43.360912 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359900 5044 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:24:43.360912 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359902 5044 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:24:43.360912 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359905 5044 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:24:43.360912 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359907 5044 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:24:43.360912 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359910 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:24:43.360912 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359913 5044 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:24:43.360912 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359916 5044 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:24:43.360912 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359918 5044 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:24:43.360912 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359921 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:24:43.360912 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359924 5044 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:24:43.360912 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359926 5044 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:24:43.360912 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359929 5044 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:24:43.360912 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359931 5044 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:24:43.360912 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359934 5044 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:24:43.360912 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359938 5044 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:24:43.361376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359942 5044 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:24:43.361376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359945 5044 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:24:43.361376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359948 5044 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:24:43.361376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359951 5044 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:24:43.361376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359954 5044 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:24:43.361376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359958 5044 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:24:43.361376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359961 5044 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:24:43.361376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359965 5044 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:24:43.361376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359968 5044 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:24:43.361376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359971 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:24:43.361376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359973 5044 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:24:43.361376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359976 5044 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:24:43.361376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359979 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:24:43.361376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359981 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:24:43.361376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359984 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:24:43.361376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359986 5044 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:24:43.361376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359989 5044 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:24:43.361376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359992 5044 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:24:43.361376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359994 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:24:43.361885 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.359998 5044 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:24:43.361885 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360001 5044 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:24:43.361885 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360003 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:24:43.361885 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360006 5044 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:24:43.361885 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360008 5044 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:24:43.361885 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360011 5044 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:24:43.361885 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360013 5044 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:24:43.361885 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360016 5044 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:24:43.361885 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360018 5044 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:24:43.361885 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360022 5044 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:24:43.361885 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360024 5044 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:24:43.361885 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360027 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:24:43.361885 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360030 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:24:43.361885 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360032 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:24:43.361885 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360035 5044 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:24:43.361885 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360037 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:24:43.361885 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360040 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:24:43.361885 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360042 5044 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:24:43.361885 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360045 5044 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:24:43.362384 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360048 5044 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:24:43.362384 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360050 5044 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:24:43.362384 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360053 5044 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:24:43.362384 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360056 5044 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:24:43.362384 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360063 5044 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:24:43.362384 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360802 5044 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:24:43.362384 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360809 5044 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:24:43.362384 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360812 5044 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:24:43.362384 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360815 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:24:43.362384 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360818 5044 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:24:43.362384 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360821 5044 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:24:43.362384 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360824 5044 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:24:43.362384 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360826 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:24:43.362384 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360829 5044 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:24:43.362384 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360832 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:24:43.362384 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360835 5044 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:24:43.362384 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360838 5044 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:24:43.362384 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360841 5044 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:24:43.362384 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360844 5044 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:24:43.362865 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360846 5044 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:24:43.362865 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360849 5044 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:24:43.362865 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360852 5044 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:24:43.362865 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360855 5044 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:24:43.362865 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360857 5044 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:24:43.362865 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360860 5044 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:24:43.362865 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360863 5044 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:24:43.362865 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360865 5044 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:24:43.362865 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360868 5044 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:24:43.362865 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360871 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:24:43.362865 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360873 5044 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:24:43.362865 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360875 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:24:43.362865 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360879 5044 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:24:43.362865 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360882 5044 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:24:43.362865 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360884 5044 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:24:43.362865 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360887 5044 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:24:43.362865 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360889 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:24:43.362865 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360892 5044 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:24:43.362865 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360911 5044 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:24:43.362865 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360916 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:24:43.363472 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360919 5044 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:24:43.363472 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360922 5044 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:24:43.363472 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360925 5044 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:24:43.363472 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360928 5044 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:24:43.363472 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360931 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:24:43.363472 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360933 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:24:43.363472 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360936 5044 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:24:43.363472 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360940 5044 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:24:43.363472 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360944 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:24:43.363472 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360948 5044 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:24:43.363472 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360950 5044 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:24:43.363472 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360953 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:24:43.363472 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360956 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:24:43.363472 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360958 5044 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:24:43.363472 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360961 5044 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:24:43.363472 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360964 5044 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:24:43.363472 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360967 5044 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:24:43.363472 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360969 5044 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:24:43.363472 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360972 5044 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:24:43.363952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360974 5044 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:24:43.363952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360977 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:24:43.363952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360979 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:24:43.363952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360982 5044 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:24:43.363952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360985 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:24:43.363952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360987 5044 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:24:43.363952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360990 5044 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:24:43.363952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360993 5044 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:24:43.363952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360995 5044 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:24:43.363952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.360998 5044 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:24:43.363952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361002 5044 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:24:43.363952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361005 5044 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:24:43.363952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361009 5044 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:24:43.363952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361012 5044 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:24:43.363952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361014 5044 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:24:43.363952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361017 5044 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:24:43.363952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361019 5044 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:24:43.363952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361022 5044 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:24:43.363952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361024 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:24:43.364628 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361027 5044 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:24:43.364628 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361029 5044 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:24:43.364628 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361032 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:24:43.364628 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361034 5044 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:24:43.364628 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361037 5044 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:24:43.364628 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361040 5044 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:24:43.364628 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361043 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:24:43.364628 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361045 5044 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:24:43.364628 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361048 5044 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:24:43.364628 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361050 5044 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:24:43.364628 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361053 5044 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:24:43.364628 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361055 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:24:43.364628 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361058 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:24:43.364628 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361060 5044 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:24:43.364628 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361063 5044 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:24:43.364628 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361065 5044 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:24:43.364628 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361068 5044 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:24:43.364628 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361070 5044 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:24:43.364628 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361073 5044 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:24:43.364628 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361076 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:24:43.365554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361078 5044 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:24:43.365554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361081 5044 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:24:43.365554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361083 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:24:43.365554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361086 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:24:43.365554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361088 5044 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:24:43.365554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361091 5044 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:24:43.365554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361099 5044 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:24:43.365554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361102 5044 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:24:43.365554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361104 5044 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:24:43.365554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361108 5044 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:24:43.365554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361116 5044 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:24:43.365554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361120 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:24:43.365554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361122 5044 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:24:43.365554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361125 5044 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:24:43.365554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361599 5044 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:24:43.365554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361605 5044 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:24:43.365554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361608 5044 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:24:43.365554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361610 5044 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:24:43.365554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361613 5044 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:24:43.365554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361616 5044 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:24:43.366314 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361619 5044 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:24:43.366314 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361623 5044 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:24:43.366314 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361627 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:24:43.366314 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361630 5044 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:24:43.366314 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361633 5044 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:24:43.366314 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361636 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:24:43.366314 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361638 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:24:43.366314 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361641 5044 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:24:43.366314 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361643 5044 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:24:43.366314 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361646 5044 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:24:43.366314 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361648 5044 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:24:43.366314 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361651 5044 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:24:43.366314 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361654 5044 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:24:43.366314 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361656 5044 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:24:43.366314 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361659 5044 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:24:43.366314 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361662 5044 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:24:43.366314 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361664 5044 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:24:43.366314 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361682 5044 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:24:43.366314 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361694 5044 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:24:43.366314 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361697 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:24:43.366876 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361700 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:24:43.366876 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361703 5044 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:24:43.366876 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361706 5044 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:24:43.366876 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361709 5044 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:24:43.366876 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361711 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:24:43.366876 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361714 5044 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:24:43.366876 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361716 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:24:43.366876 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361719 5044 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:24:43.366876 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361722 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:24:43.366876 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361724 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:24:43.366876 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361727 5044 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:24:43.366876 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361730 5044 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:24:43.366876 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361732 5044 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:24:43.366876 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361735 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:24:43.366876 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361737 5044 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:24:43.366876 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361740 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:24:43.366876 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361743 5044 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:24:43.366876 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361745 5044 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:24:43.366876 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361748 5044 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:24:43.366876 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361751 5044 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:24:43.367376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361754 5044 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:24:43.367376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361757 5044 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:24:43.367376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361760 5044 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:24:43.367376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361762 5044 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:24:43.367376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361765 5044 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:24:43.367376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361768 5044 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:24:43.367376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361770 5044 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:24:43.367376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361773 5044 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:24:43.367376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361776 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:24:43.367376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361779 5044 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:24:43.367376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361781 5044 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:24:43.367376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361784 5044 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:24:43.367376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361788 5044 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:24:43.367376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361792 5044 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:24:43.367376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361795 5044 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:24:43.367376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361798 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:24:43.367376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361801 5044 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:24:43.367376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361804 5044 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:24:43.367376 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361806 5044 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:24:43.367903 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361809 5044 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:24:43.367903 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361811 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:24:43.367903 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361816 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:24:43.367903 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361819 5044 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:24:43.367903 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361821 5044 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:24:43.367903 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361824 5044 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:24:43.367903 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361827 5044 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:24:43.367903 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361829 5044 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:24:43.367903 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361832 5044 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:24:43.367903 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361834 5044 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:24:43.367903 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361838 5044 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:24:43.367903 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361841 5044 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:24:43.367903 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361843 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:24:43.367903 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361846 5044 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:24:43.367903 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361848 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:24:43.367903 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361851 5044 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:24:43.367903 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361853 5044 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:24:43.367903 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361856 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:24:43.367903 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361859 5044 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:24:43.367903 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361861 5044 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:24:43.368620 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361864 5044 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:24:43.368620 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361866 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:24:43.368620 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361869 5044 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:24:43.368620 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361871 5044 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:24:43.368620 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361873 5044 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:24:43.368620 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361877 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:24:43.368620 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361880 5044 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:24:43.368620 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361882 5044 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:24:43.368620 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361885 5044 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:24:43.368620 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361887 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:24:43.368620 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361890 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:24:43.368620 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361893 5044 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:24:43.368620 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361896 5044 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:24:43.368620 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361898 5044 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:24:43.368620 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361902 5044 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:24:43.368620 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361905 5044 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:24:43.368620 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361907 5044 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:24:43.368620 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361913 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:24:43.368620 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361916 5044 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361919 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.361921 5044 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362015 5044 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362023 5044 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362028 5044 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362033 5044 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362037 5044 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362041 5044 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362048 5044 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362052 5044 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362055 5044 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362058 5044 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362062 5044 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362065 5044 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362068 5044 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362071 5044 flags.go:64] FLAG: --cgroup-root="" Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362074 5044 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362077 5044 flags.go:64] FLAG: --client-ca-file="" Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362080 5044 flags.go:64] FLAG: --cloud-provider="external" Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362084 5044 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362088 5044 flags.go:64] FLAG: --cluster-domain="" Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362091 5044 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 01:24:43.369505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362094 5044 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362098 5044 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362101 5044 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362105 5044 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362108 5044 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362113 5044 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362116 5044 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362119 5044 flags.go:64] FLAG: --contention-profiling="false" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362123 5044 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362127 5044 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362130 5044 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362133 5044 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362137 5044 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362140 5044 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362143 5044 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362146 5044 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362149 5044 flags.go:64] FLAG: --enable-server="true" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362152 5044 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362155 5044 flags.go:64] FLAG: --event-burst="100" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362159 5044 flags.go:64] FLAG: --event-qps="50" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362162 5044 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362164 5044 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362168 5044 flags.go:64] FLAG: --eviction-hard="" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362172 5044 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 01:24:43.370353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362175 5044 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362178 5044 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362181 5044 flags.go:64] FLAG: --eviction-soft="" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362184 5044 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362188 5044 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362190 5044 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362194 5044 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362197 5044 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362200 5044 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362203 5044 flags.go:64] FLAG: --feature-gates="" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362207 5044 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362210 5044 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362214 5044 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362218 5044 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362221 5044 flags.go:64] FLAG: --healthz-port="10248" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362224 5044 flags.go:64] FLAG: --help="false" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362228 5044 flags.go:64] FLAG: --hostname-override="ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362232 5044 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362235 5044 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362238 5044 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362241 5044 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362245 5044 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362248 5044 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 01:24:43.370988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362258 5044 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362262 5044 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362265 5044 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362268 5044 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362271 5044 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362274 5044 flags.go:64] FLAG: --kube-reserved="" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362277 5044 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362280 5044 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362283 5044 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362286 5044 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362289 5044 flags.go:64] FLAG: --lock-file="" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362292 5044 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362295 5044 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362298 5044 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362303 5044 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362306 5044 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362310 5044 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362313 5044 flags.go:64] FLAG: --logging-format="text" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362316 5044 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362319 5044 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362322 5044 flags.go:64] FLAG: --manifest-url="" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362325 5044 flags.go:64] FLAG: --manifest-url-header="" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362332 5044 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362335 5044 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362338 5044 flags.go:64] FLAG: --max-pods="110" Nov 02 01:24:43.371598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362342 5044 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362345 5044 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362349 5044 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362352 5044 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362355 5044 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362358 5044 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362361 5044 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362368 5044 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362371 5044 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362375 5044 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362378 5044 flags.go:64] FLAG: --pod-cidr="" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362381 5044 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362384 5044 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362387 5044 flags.go:64] FLAG: --pods-per-core="0" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362390 5044 flags.go:64] FLAG: --port="10250" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362393 5044 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362396 5044 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-071a45d511e7feffe" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362399 5044 flags.go:64] FLAG: --qos-reserved="" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362402 5044 flags.go:64] FLAG: --read-only-port="10255" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362405 5044 flags.go:64] FLAG: --register-node="true" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362408 5044 flags.go:64] FLAG: --register-with-taints="" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362412 5044 flags.go:64] FLAG: --registry-burst="10" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362415 5044 flags.go:64] FLAG: --registry-qps="5" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362418 5044 flags.go:64] FLAG: --reserved-cpus="" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362421 5044 flags.go:64] FLAG: --reserved-memory="" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362425 5044 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 01:24:43.372291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362428 5044 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362431 5044 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362434 5044 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362438 5044 flags.go:64] FLAG: --runonce="false" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362441 5044 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362445 5044 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362449 5044 flags.go:64] FLAG: --seccomp-default="false" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362452 5044 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362455 5044 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362458 5044 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362461 5044 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362465 5044 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362467 5044 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362470 5044 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362473 5044 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362477 5044 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362480 5044 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362483 5044 flags.go:64] FLAG: --system-cgroups="" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362486 5044 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362491 5044 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362494 5044 flags.go:64] FLAG: --tls-cert-file="" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362497 5044 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362501 5044 flags.go:64] FLAG: --tls-min-version="" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362504 5044 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362507 5044 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 01:24:43.372947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362509 5044 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362512 5044 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362515 5044 flags.go:64] FLAG: --v="2" Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362519 5044 flags.go:64] FLAG: --version="false" Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362523 5044 flags.go:64] FLAG: --vmodule="" Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362527 5044 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.362531 5044 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362688 5044 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362694 5044 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362698 5044 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362702 5044 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362705 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362708 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362711 5044 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362714 5044 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362717 5044 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362719 5044 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362722 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362725 5044 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362727 5044 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362730 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:24:43.373546 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362732 5044 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:24:43.374207 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362735 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:24:43.374207 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362738 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:24:43.374207 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362740 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:24:43.374207 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362743 5044 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:24:43.374207 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362745 5044 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:24:43.374207 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362748 5044 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:24:43.374207 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362751 5044 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:24:43.374207 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362753 5044 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:24:43.374207 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362756 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:24:43.374207 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362759 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:24:43.374207 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362761 5044 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:24:43.374207 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362763 5044 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:24:43.374207 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362766 5044 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:24:43.374207 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362768 5044 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:24:43.374207 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362771 5044 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:24:43.374207 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362774 5044 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:24:43.374207 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362776 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:24:43.374207 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362783 5044 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:24:43.374207 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362786 5044 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:24:43.374207 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362789 5044 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:24:43.374952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362793 5044 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:24:43.374952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362795 5044 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:24:43.374952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362798 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:24:43.374952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362801 5044 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:24:43.374952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362804 5044 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:24:43.374952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362807 5044 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:24:43.374952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362809 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:24:43.374952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362812 5044 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:24:43.374952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362814 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:24:43.374952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362817 5044 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:24:43.374952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362820 5044 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:24:43.374952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362822 5044 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:24:43.374952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362825 5044 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:24:43.374952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362827 5044 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:24:43.374952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362830 5044 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:24:43.374952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362833 5044 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:24:43.374952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362835 5044 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:24:43.374952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362838 5044 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:24:43.374952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362840 5044 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:24:43.374952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362843 5044 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:24:43.375562 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362845 5044 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:24:43.375562 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362848 5044 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:24:43.375562 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362850 5044 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:24:43.375562 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362853 5044 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:24:43.375562 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362856 5044 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:24:43.375562 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362859 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:24:43.375562 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362861 5044 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:24:43.375562 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362864 5044 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:24:43.375562 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362866 5044 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:24:43.375562 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362869 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:24:43.375562 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362872 5044 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:24:43.375562 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362875 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:24:43.375562 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362878 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:24:43.375562 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362881 5044 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:24:43.375562 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362884 5044 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:24:43.375562 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362887 5044 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:24:43.375562 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362890 5044 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:24:43.375562 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362893 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:24:43.375562 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362895 5044 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:24:43.376146 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362898 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:24:43.376146 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362901 5044 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:24:43.376146 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362905 5044 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:24:43.376146 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362909 5044 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:24:43.376146 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362913 5044 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:24:43.376146 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362916 5044 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:24:43.376146 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362922 5044 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:24:43.376146 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362928 5044 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:24:43.376146 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362933 5044 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:24:43.376146 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362937 5044 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:24:43.376146 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362941 5044 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:24:43.376146 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362945 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:24:43.376146 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362948 5044 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:24:43.376146 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362952 5044 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:24:43.376146 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362956 5044 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:24:43.376146 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362959 5044 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:24:43.376146 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362963 5044 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:24:43.376146 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362967 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:24:43.376146 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362971 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:24:43.376601 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362975 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:24:43.376601 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362978 5044 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:24:43.376601 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362983 5044 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:24:43.376601 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362987 5044 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:24:43.376601 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362992 5044 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:24:43.376601 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.362999 5044 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:24:43.376601 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363003 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:24:43.376601 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363008 5044 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:24:43.376601 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363012 5044 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:24:43.376601 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363018 5044 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:24:43.376601 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363022 5044 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:24:43.376601 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363026 5044 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:24:43.376601 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363030 5044 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:24:43.376601 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.363070 5044 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 01:24:43.376601 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363147 5044 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:24:43.376601 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363156 5044 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:24:43.377185 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363162 5044 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:24:43.377185 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363167 5044 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:24:43.377185 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363171 5044 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:24:43.377185 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363176 5044 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:24:43.377185 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363180 5044 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:24:43.377185 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363184 5044 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:24:43.377185 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363189 5044 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:24:43.377185 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363193 5044 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:24:43.377185 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363197 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:24:43.377185 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363201 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:24:43.377185 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363205 5044 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:24:43.377185 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363209 5044 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:24:43.377185 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363213 5044 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:24:43.377185 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363217 5044 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:24:43.377185 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363221 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:24:43.377185 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363225 5044 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:24:43.377185 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363229 5044 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:24:43.377185 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363233 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:24:43.377185 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363238 5044 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:24:43.377185 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363241 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:24:43.377891 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363245 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:24:43.377891 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363250 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:24:43.377891 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363255 5044 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:24:43.377891 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363260 5044 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:24:43.377891 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363264 5044 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:24:43.377891 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363268 5044 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:24:43.377891 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363274 5044 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:24:43.377891 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363278 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:24:43.377891 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363282 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:24:43.377891 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363286 5044 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:24:43.377891 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363290 5044 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:24:43.377891 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363294 5044 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:24:43.377891 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363298 5044 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:24:43.377891 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363303 5044 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:24:43.377891 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363307 5044 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:24:43.377891 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363311 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:24:43.377891 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363315 5044 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:24:43.377891 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363319 5044 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:24:43.377891 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363322 5044 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:24:43.377891 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363326 5044 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:24:43.378520 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363330 5044 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:24:43.378520 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363335 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:24:43.378520 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363339 5044 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:24:43.378520 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363343 5044 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:24:43.378520 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363347 5044 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:24:43.378520 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363351 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:24:43.378520 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363356 5044 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:24:43.378520 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363360 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:24:43.378520 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363364 5044 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:24:43.378520 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363368 5044 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:24:43.378520 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363373 5044 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:24:43.378520 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363376 5044 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:24:43.378520 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363380 5044 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:24:43.378520 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363384 5044 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:24:43.378520 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363388 5044 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:24:43.378520 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363393 5044 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:24:43.378520 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363397 5044 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:24:43.378520 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363401 5044 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:24:43.378520 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363409 5044 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:24:43.378520 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363414 5044 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:24:43.379019 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363418 5044 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:24:43.379019 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363422 5044 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:24:43.379019 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363426 5044 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:24:43.379019 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363430 5044 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:24:43.379019 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363435 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:24:43.379019 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363439 5044 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:24:43.379019 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363443 5044 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:24:43.379019 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363447 5044 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:24:43.379019 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363451 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:24:43.379019 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363455 5044 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:24:43.379019 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363459 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:24:43.379019 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363463 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:24:43.379019 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363468 5044 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:24:43.379019 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363472 5044 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:24:43.379019 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363476 5044 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:24:43.379019 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363480 5044 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:24:43.379019 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363484 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:24:43.379019 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363489 5044 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:24:43.379019 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363493 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:24:43.379476 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363497 5044 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:24:43.379476 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363501 5044 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:24:43.379476 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363505 5044 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:24:43.379476 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363509 5044 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:24:43.379476 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363513 5044 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:24:43.379476 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363519 5044 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:24:43.379476 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363528 5044 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:24:43.379476 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363533 5044 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:24:43.379476 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363537 5044 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:24:43.379476 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363543 5044 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:24:43.379476 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363548 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:24:43.379476 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363554 5044 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:24:43.379476 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363558 5044 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:24:43.379476 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363562 5044 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:24:43.379476 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363567 5044 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:24:43.379476 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363571 5044 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:24:43.379476 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363575 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:24:43.379476 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363580 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:24:43.379476 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363584 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:24:43.379988 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363589 5044 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:24:43.379988 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363593 5044 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:24:43.379988 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363597 5044 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:24:43.379988 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363604 5044 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:24:43.379988 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363610 5044 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:24:43.379988 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.363615 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:24:43.379988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.368666 5044 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 01:24:43.379988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.368712 5044 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 01:24:43.379988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.368737 5044 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 01:24:43.379988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.368745 5044 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 01:24:43.379988 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.368979 5044 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:24:43.379988 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369007 5044 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:24:43.379988 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369015 5044 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:24:43.379988 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369023 5044 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:24:43.379988 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369031 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:24:43.379988 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369037 5044 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:24:43.379988 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369042 5044 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:24:43.379988 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369053 5044 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:24:43.379988 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369057 5044 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:24:43.380471 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369061 5044 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:24:43.380471 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369066 5044 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:24:43.380471 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369089 5044 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:24:43.380471 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369097 5044 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:24:43.380471 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369102 5044 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:24:43.380471 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369107 5044 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:24:43.380471 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369111 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:24:43.380471 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369116 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:24:43.380471 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369121 5044 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:24:43.380471 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369126 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:24:43.380471 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369131 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:24:43.380471 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369140 5044 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:24:43.380471 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369145 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:24:43.380471 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369166 5044 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:24:43.380471 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369173 5044 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:24:43.380471 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369177 5044 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:24:43.380471 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369182 5044 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:24:43.380471 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369187 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:24:43.380471 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369191 5044 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:24:43.380471 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369196 5044 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:24:43.381051 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369201 5044 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:24:43.381051 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369207 5044 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:24:43.381051 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369211 5044 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:24:43.381051 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369220 5044 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:24:43.381051 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369225 5044 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:24:43.381051 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369248 5044 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:24:43.381051 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369254 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:24:43.381051 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369258 5044 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:24:43.381051 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369263 5044 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:24:43.381051 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369267 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:24:43.381051 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369272 5044 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:24:43.381051 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369276 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:24:43.381051 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369281 5044 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:24:43.381051 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369286 5044 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:24:43.381051 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369291 5044 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:24:43.381051 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369295 5044 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:24:43.381051 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369306 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:24:43.381051 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369320 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:24:43.381051 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369325 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:24:43.381051 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369329 5044 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:24:43.381755 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369335 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:24:43.381755 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369340 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:24:43.381755 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369344 5044 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:24:43.381755 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369351 5044 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:24:43.381755 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369357 5044 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:24:43.381755 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369362 5044 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:24:43.381755 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369366 5044 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:24:43.381755 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369371 5044 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:24:43.381755 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369381 5044 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:24:43.381755 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369386 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:24:43.381755 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369391 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:24:43.381755 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369395 5044 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:24:43.381755 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369400 5044 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:24:43.381755 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369404 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:24:43.381755 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369409 5044 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:24:43.381755 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369413 5044 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:24:43.381755 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369418 5044 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:24:43.381755 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369423 5044 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:24:43.381755 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369428 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:24:43.382457 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369432 5044 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:24:43.382457 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369437 5044 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:24:43.382457 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369445 5044 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:24:43.382457 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369450 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:24:43.382457 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369454 5044 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:24:43.382457 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369459 5044 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:24:43.382457 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369463 5044 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:24:43.382457 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369467 5044 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:24:43.382457 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369472 5044 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:24:43.382457 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369477 5044 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:24:43.382457 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369483 5044 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:24:43.382457 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369488 5044 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:24:43.382457 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369492 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:24:43.382457 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369497 5044 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:24:43.382457 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369501 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:24:43.382457 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369509 5044 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:24:43.382457 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369513 5044 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:24:43.382457 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369518 5044 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:24:43.382457 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369522 5044 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:24:43.382457 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369527 5044 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:24:43.383076 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369531 5044 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:24:43.383076 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369535 5044 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:24:43.383076 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369540 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:24:43.383076 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369544 5044 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:24:43.383076 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369549 5044 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:24:43.383076 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369553 5044 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:24:43.383076 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369558 5044 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:24:43.383076 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369563 5044 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:24:43.383076 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369572 5044 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:24:43.383076 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369577 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:24:43.383076 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369581 5044 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:24:43.383076 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369586 5044 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:24:43.383076 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369590 5044 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:24:43.383076 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369595 5044 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:24:43.383076 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369599 5044 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:24:43.383076 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369604 5044 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:24:43.383076 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369609 5044 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:24:43.383076 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.369613 5044 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:24:43.383498 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.369722 5044 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 01:24:43.383498 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370162 5044 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:24:43.383498 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370172 5044 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:24:43.383498 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370175 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:24:43.383498 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370179 5044 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:24:43.383498 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370182 5044 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:24:43.383498 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370185 5044 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:24:43.383498 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370188 5044 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:24:43.383498 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370192 5044 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:24:43.383498 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370194 5044 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:24:43.383498 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370197 5044 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:24:43.383498 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370200 5044 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:24:43.383498 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370203 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:24:43.383498 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370208 5044 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:24:43.383498 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370211 5044 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:24:43.383498 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370214 5044 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:24:43.383980 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370216 5044 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:24:43.383980 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370219 5044 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:24:43.383980 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370222 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:24:43.383980 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370225 5044 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:24:43.383980 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370228 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:24:43.383980 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370230 5044 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:24:43.383980 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370233 5044 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:24:43.383980 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370235 5044 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:24:43.383980 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370238 5044 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:24:43.383980 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370241 5044 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:24:43.383980 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370245 5044 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:24:43.383980 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370249 5044 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:24:43.383980 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370262 5044 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:24:43.383980 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370265 5044 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:24:43.383980 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370267 5044 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:24:43.383980 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370270 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:24:43.383980 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370273 5044 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:24:43.383980 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370275 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:24:43.383980 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370278 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:24:43.384466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370280 5044 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:24:43.384466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370283 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:24:43.384466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370285 5044 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:24:43.384466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370288 5044 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:24:43.384466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370291 5044 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:24:43.384466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370293 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:24:43.384466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370296 5044 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:24:43.384466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370299 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:24:43.384466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370301 5044 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:24:43.384466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370304 5044 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:24:43.384466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370308 5044 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:24:43.384466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370310 5044 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:24:43.384466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370313 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:24:43.384466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370316 5044 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:24:43.384466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370318 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:24:43.384466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370321 5044 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:24:43.384466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370324 5044 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:24:43.384466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370326 5044 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:24:43.384466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370329 5044 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:24:43.384466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370332 5044 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:24:43.384973 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370334 5044 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:24:43.384973 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370337 5044 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:24:43.384973 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370339 5044 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:24:43.384973 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370342 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:24:43.384973 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370345 5044 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:24:43.384973 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370347 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:24:43.384973 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370350 5044 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:24:43.384973 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370352 5044 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:24:43.384973 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370355 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:24:43.384973 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370358 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:24:43.384973 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370360 5044 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:24:43.384973 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370363 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:24:43.384973 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370366 5044 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:24:43.384973 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370368 5044 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:24:43.384973 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370371 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:24:43.384973 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370374 5044 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:24:43.384973 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370377 5044 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:24:43.384973 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370379 5044 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:24:43.384973 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370383 5044 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:24:43.385435 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370388 5044 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:24:43.385435 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370391 5044 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:24:43.385435 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370394 5044 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:24:43.385435 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370398 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:24:43.385435 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370400 5044 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:24:43.385435 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370403 5044 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:24:43.385435 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370406 5044 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:24:43.385435 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370408 5044 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:24:43.385435 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370410 5044 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:24:43.385435 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370413 5044 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:24:43.385435 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370416 5044 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:24:43.385435 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370418 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:24:43.385435 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370421 5044 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:24:43.385435 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370423 5044 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:24:43.385435 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370426 5044 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:24:43.385435 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370429 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:24:43.385435 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370431 5044 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:24:43.385435 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370434 5044 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:24:43.385435 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370437 5044 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:24:43.385435 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370439 5044 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:24:43.386288 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370442 5044 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:24:43.386288 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370444 5044 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:24:43.386288 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370447 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:24:43.386288 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370449 5044 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:24:43.386288 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370452 5044 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:24:43.386288 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370454 5044 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:24:43.386288 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370457 5044 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:24:43.386288 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370459 5044 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:24:43.386288 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370462 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:24:43.386288 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370465 5044 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:24:43.386288 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370467 5044 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:24:43.386288 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370470 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:24:43.386288 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370473 5044 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:24:43.386288 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.370500 5044 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 01:24:43.386288 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370536 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 01:24:43.386288 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370541 5044 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 01:24:43.386902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370544 5044 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 01:24:43.386902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370547 5044 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 01:24:43.386902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370550 5044 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 01:24:43.386902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370552 5044 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 01:24:43.386902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370555 5044 feature_gate.go:386] unrecognized feature gate: Example Nov 02 01:24:43.386902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370558 5044 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 01:24:43.386902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370561 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 01:24:43.386902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370563 5044 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 01:24:43.386902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370565 5044 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 01:24:43.386902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370568 5044 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 01:24:43.386902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370570 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 01:24:43.386902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370573 5044 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 01:24:43.386902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370576 5044 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 01:24:43.386902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370579 5044 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 01:24:43.386902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370582 5044 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 01:24:43.386902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370584 5044 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 01:24:43.386902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370588 5044 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 01:24:43.386902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370591 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 01:24:43.386902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370594 5044 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 01:24:43.386902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370596 5044 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 01:24:43.387380 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370599 5044 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 01:24:43.387380 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370602 5044 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 01:24:43.387380 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370604 5044 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 01:24:43.387380 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370607 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 01:24:43.387380 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370609 5044 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 01:24:43.387380 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370612 5044 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 01:24:43.387380 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370614 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 01:24:43.387380 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370617 5044 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 01:24:43.387380 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370625 5044 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 01:24:43.387380 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370628 5044 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 01:24:43.387380 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370631 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 01:24:43.387380 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370635 5044 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 01:24:43.387380 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370638 5044 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 01:24:43.387380 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370640 5044 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 01:24:43.387380 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370643 5044 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 01:24:43.387380 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370646 5044 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 01:24:43.387380 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370648 5044 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 01:24:43.387380 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370651 5044 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 01:24:43.387380 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370654 5044 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 01:24:43.387893 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370656 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 01:24:43.387893 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370659 5044 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 01:24:43.387893 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370661 5044 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 01:24:43.387893 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370664 5044 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 01:24:43.387893 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370687 5044 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 01:24:43.387893 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370692 5044 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 01:24:43.387893 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370696 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 01:24:43.387893 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370698 5044 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 01:24:43.387893 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370701 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 01:24:43.387893 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370705 5044 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 01:24:43.387893 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370708 5044 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 01:24:43.387893 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370710 5044 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 01:24:43.387893 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370713 5044 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 01:24:43.387893 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370716 5044 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 01:24:43.387893 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370721 5044 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 01:24:43.387893 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370725 5044 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 01:24:43.387893 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370728 5044 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 01:24:43.387893 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370730 5044 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 01:24:43.387893 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370733 5044 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 01:24:43.388350 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370736 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 01:24:43.388350 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370738 5044 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 01:24:43.388350 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370741 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 01:24:43.388350 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370745 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 01:24:43.388350 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370748 5044 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 01:24:43.388350 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370752 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 01:24:43.388350 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370756 5044 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 01:24:43.388350 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370760 5044 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 01:24:43.388350 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370764 5044 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 01:24:43.388350 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370768 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 01:24:43.388350 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370771 5044 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 01:24:43.388350 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370774 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 01:24:43.388350 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370776 5044 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 01:24:43.388350 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370779 5044 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 01:24:43.388350 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370781 5044 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 01:24:43.388350 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370784 5044 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 01:24:43.388350 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370787 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 01:24:43.388350 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370789 5044 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 01:24:43.388350 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370792 5044 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 01:24:43.388350 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370794 5044 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 01:24:43.388839 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370797 5044 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 01:24:43.388839 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370800 5044 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 01:24:43.388839 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370804 5044 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 01:24:43.388839 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370806 5044 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 01:24:43.388839 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370809 5044 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 01:24:43.388839 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370811 5044 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 01:24:43.388839 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370814 5044 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 01:24:43.388839 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370817 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 01:24:43.388839 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370819 5044 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 01:24:43.388839 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370822 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 01:24:43.388839 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370825 5044 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 01:24:43.388839 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370827 5044 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 01:24:43.388839 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370830 5044 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 01:24:43.388839 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370832 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 01:24:43.388839 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370835 5044 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 01:24:43.388839 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370837 5044 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 01:24:43.388839 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370841 5044 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 01:24:43.388839 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370846 5044 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 01:24:43.388839 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370848 5044 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 01:24:43.389286 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370851 5044 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 01:24:43.389286 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370854 5044 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 01:24:43.389286 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370856 5044 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 01:24:43.389286 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370860 5044 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 01:24:43.389286 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370864 5044 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 01:24:43.389286 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370867 5044 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 01:24:43.389286 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:43.370871 5044 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 01:24:43.389286 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.370929 5044 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 01:24:43.389286 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.371850 5044 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 01:24:43.389286 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.371896 5044 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 01:24:43.389286 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.372064 5044 server.go:1014] "Starting client certificate rotation" Nov 02 01:24:43.389286 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.372213 5044 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 01:24:43.389286 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.372254 5044 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 01:19:00 +0000 UTC" deadline="2028-07-04 19:48:22.801970023 +0000 UTC" Nov 02 01:24:43.389286 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.372286 5044 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14658h23m39.429686979s" Nov 02 01:24:43.389286 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.372504 5044 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 01:24:43.389650 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.373573 5044 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 01:24:43.389650 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.375061 5044 log.go:25] "Validated CRI v1 runtime API" Nov 02 01:24:43.389650 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.376805 5044 log.go:25] "Validated CRI v1 image API" Nov 02 01:24:43.389650 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.377573 5044 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 01:24:43.389650 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.379731 5044 fs.go:135] Filesystem UUIDs: map[488f296c-96e6-49a6-af0f-103489929e55:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d9515f3b-97e5-460a-9d3f-0f82133e4c90:/dev/nvme0n1p3] Nov 02 01:24:43.389939 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.379752 5044 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06f2e60e44f6b1ae0d0ce0b7cdac42b43ef1a76ff92b2a849246f46e15896d45/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06f2e60e44f6b1ae0d0ce0b7cdac42b43ef1a76ff92b2a849246f46e15896d45/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a6d9848ff171086e151f397491a8f514dc8832592a239c33122c3284875456f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a6d9848ff171086e151f397491a8f514dc8832592a239c33122c3284875456f/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f2e12d4751d7b88c57875fc67990526ffd54c6b50370cda54f8241789bdf974/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f2e12d4751d7b88c57875fc67990526ffd54c6b50370cda54f8241789bdf974/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/194ec6187c7b572797fd8f78e02cb66a027fb1b3f1f870aa4adda003f033d9e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/194ec6187c7b572797fd8f78e02cb66a027fb1b3f1f870aa4adda003f033d9e7/userdata/shm major:0 minor:322 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/270cd97377d77588e4c335e61b9b1eba3f5ea162f96bc871d31f4f2e1213cc10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/270cd97377d77588e4c335e61b9b1eba3f5ea162f96bc871d31f4f2e1213cc10/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c4476138a7bf2989aac4600b621afb29324711330f782eb69f9bbce13d98075/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c4476138a7bf2989aac4600b621afb29324711330f782eb69f9bbce13d98075/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d604ba2867e44ccc9b62c8fe569b770ac1a144bca0d2ddfedade3e526a2ff0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d604ba2867e44ccc9b62c8fe569b770ac1a144bca0d2ddfedade3e526a2ff0b/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/411ae169b7770debda9684191543d54f36415adc6398062ead31c1100c6c1d81/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/411ae169b7770debda9684191543d54f36415adc6398062ead31c1100c6c1d81/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e417561d9def4122cb070f5244c8735d39aa9d61f3ac1d114ff9a4e5e5c4a4f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e417561d9def4122cb070f5244c8735d39aa9d61f3ac1d114ff9a4e5e5c4a4f/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b45cf67f3d40c4644ea1e2a59eeb281e816e65de1f906142311859c233cb806/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b45cf67f3d40c4644ea1e2a59eeb281e816e65de1f906142311859c233cb806/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60ee38920eb119bf15b637d23b79c11689392dd6adbec04062c5121447e03e03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60ee38920eb119bf15b637d23b79c11689392dd6adbec04062c5121447e03e03/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/624e8843c916c2fcc231815c561ff5315e6251132d93db11dbffb2278bb17833/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/624e8843c916c2fcc231815c561ff5315e6251132d93db11dbffb2278bb17833/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/671ea2856c9b53a7db137f865f26d6585cd38fb6098f982fdcc8526857b37f9f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/671ea2856c9b53a7db137f865f26d6585cd38fb6098f982fdcc8526857b37f9f/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/780e6524fb0d75e94a6e9d5e7000be249c6dc9f4a2fdae31a49b93c800814f62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/780e6524fb0d75e94a6e9d5e7000be249c6dc9f4a2fdae31a49b93c800814f62/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d2fee6e42afcc68e2cbe0d970b893f2203dbf493985ccb97070845466de3e72/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d2fee6e42afcc68e2cbe0d970b893f2203dbf493985ccb97070845466de3e72/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9fcaa49884d943ea43a86669a9512e72925c2d85280c94b971f08c50e553d79f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9fcaa49884d943ea43a86669a9512e72925c2d85280c94b971f08c50e553d79f/userdata/shm major:0 minor:339 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aef0f7510a57f5b33b937b4d573d2417f0dbf9f2f55ee57de1f762e2ea09b55d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aef0f7510a57f5b33b937b4d573d2417f0dbf9f2f55ee57de1f762e2ea09b55d/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b136b6d11358e2172ebba343dc7590986e27110dc3800b9d61c9491d619191a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b136b6d11358e2172ebba343dc7590986e27110dc3800b9d61c9491d619191a5/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cae0ce038d06591c037675c225f487723b7f02925e0eddf2839d06909619ccbb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cae0ce038d06591c037675c225f487723b7f02925e0eddf2839d06909619ccbb/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1b449089fc1b096507ccc81210b0ab22d0eef7f4816aa2fdafe9b3542bdbd4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1b449089fc1b096507ccc81210b0ab22d0eef7f4816aa2fdafe9b3542bdbd4a/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbbaa485276e9e80414df1c76559d8406a6ebf38307eccac18eda98c904bffde/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbbaa485276e9e80414df1c76559d8406a6ebf38307eccac18eda98c904bffde/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1fbaf835ff8f3a7565f0d4aefabd3fbdfac25aa4676e785fad3e60cc44e6a38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1fbaf835ff8f3a7565f0d4aefabd3fbdfac25aa4676e785fad3e60cc44e6a38/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e70bc567d899951d4b95692ad723aab0eee4d2d732c291c0b99e2ae4a2fd024a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e70bc567d899951d4b95692ad723aab0eee4d2d732c291c0b99e2ae4a2fd024a/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f28823619809a7408c348ae1039f3bde4f82f41d60899137778532d11d51718c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f28823619809a7408c348ae1039f3bde4f82f41d60899137778532d11d51718c/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb877fe27b5a235f783aeb6c121b0d7df45af00d84d72cacb0272dec49d6fd58/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb877fe27b5a235f783aeb6c121b0d7df45af00d84d72cacb0272dec49d6fd58/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ffc3d4e4c98fde42495e9d0bf5b1cc1a3a0e1b490ddd8ab72f086b1dd7fe9787/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ffc3d4e4c98fde42495e9d0bf5b1cc1a3a0e1b490ddd8ab72f086b1dd7fe9787/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/028b56de-c4b9-47c1-8bfa-761877176429/volumes/kubernetes.io~projected/kube-api-access-k4h6s:{mountpoint:/var/lib/kubelet/pods/028b56de-c4b9-47c1-8bfa-761877176429/volumes/kubernetes.io~projected/kube-api-access-k4h6s major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/028b56de-c4b9-47c1-8bfa-761877176429/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/028b56de-c4b9-47c1-8bfa-761877176429/volumes/kubernetes.io~secret/signing-key major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25437e9b-ca22-4103-ac9e-c5823f2bbbaf/volumes/kubernetes.io~projected/kube-api-access-724nj:{mountpoint:/var/lib/kubelet/pods/25437e9b-ca22-4103-ac9e-c5823f2bbbaf/volumes/kubernetes.io~projected/kube-api-access-724nj major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c8cbfce-aaf5-4362-930e-b15539b2dbc4/volumes/kubernetes.io~projected/kube-api-access-jvshf:{mountpoint:/var/lib/kubelet/pods/2c8cbfce-aaf5-4362-930e-b15539b2dbc4/volumes/kubernetes.io~projected/kube-api-access-jvshf major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e031e5c-cb8a-42b9-b334-1cbe61a9c59f/volumes/kubernetes.io~projected/kube-api-access-6z8mm:{mountpoint:/var/lib/kubelet/pods/2e031e5c-cb8a-42b9-b334-1cbe61a9c59f/volumes/kubernetes.io~projected/kube-api-access-6z8mm major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35ae4588-f166-4563-aae7-b34b5b053d0f/volumes/kubernetes.io~projected/kube-api-access-6s6ct:{mountpoint:/var/lib/kubelet/pods/35ae4588-f166-4563-aae7-b34b5b053d0f/volumes/kubernetes.io~projected/kube-api-access-6s6ct major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35ae4588-f166-4563-aae7-b34b5b053d0f/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/35ae4588-f166-4563-aae7-b34b5b053d0f/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/363dffa0-70cf-4fe0-996f-3c4d2f3b1a50/volumes/kubernetes.io~projected/kube-api-access-bxmrq:{mountpoint:/var/lib/kubelet/pods/363dffa0-70cf-4fe0-996f-3c4d2f3b1a50/volumes/kubernetes.io~projected/kube-api-access-bxmrq major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/476d365d-dd0a-41e6-a186-5e92551be763/volumes/kubernetes.io~projected/kube-api-access-dc4n8:{mountpoint:/var/lib/kubelet/pods/476d365d-dd0a-41e6-a186-5e92551be763/volumes/kubernetes.io~projected/kube-api-access-dc4n8 major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4abd5311-4a92-4ac5-8aa0-779dd8774577/volumes/kubernetes.io~projected/kube-api-access-tpwzf:{mountpoint:/var/lib/kubelet/pods/4abd5311-4a92-4ac5-8aa0-779dd8774577/volumes/kubernetes.io~projected/kube-api-access-tpwzf major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4abd5311-4a92-4ac5-8aa0-779dd8774577/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/4abd5311-4a92-4ac5-8aa0-779dd8774577/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54db4b99-9007-429a-a267-b4924a8729e6/volumes/kubernetes.io~projected/kube-api-access-57blk:{mountpoint:/var/lib/kubelet/pods/54db4b99-9007-429a-a267-b4924a8729e6/volumes/kubernetes.io~projected/kube-api-access-57blk major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59ccadf3-202d-481d-8c8b-c4fa86c92b8b/volumes/kubernetes.io~projected/kube-api-access-xp982:{mountpoint:/var/lib/kubelet/pods/59ccadf3-202d-481d-8c8b-c4fa86c92b8b/volumes/kubernetes.io~projected/kube-api-access-xp982 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99/volumes/kubernetes.io~projected/kube-api-access-jg9vq:{mountpoint:/var/lib/kubelet/pods/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99/volumes/kubernetes.io~projected/kube-api-access-jg9vq major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/673cb927-a5fc-4237-9b4c-fb196801d70c/volumes/kubernetes.io~projected/kube-api-access-xrnrh:{mountpoint:/var/lib/kubelet/pods/673cb927-a5fc-4237-9b4c-fb196801d70c/volumes/kubernetes.io~projected/kube-api-access-xrnrh major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79f97db3-814b-4849-ab8d-305dbcd3449c/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/79f97db3-814b-4849-ab8d-305dbcd3449c/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c285da3-2e11-498d-94f1-76c0ce604553/volumes/kubernetes.io~projected/kube-api-access-v8wfd:{mountpoint:/var/lib/kubelet/pods/7c285da3-2e11-498d-94f1-76c0ce604553/volumes/kubernetes.io~projected/kube-api-access-v8wfd major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c285da3-2e11-498d-94f1-76c0ce604553/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/7c285da3-2e11-498d-94f1-76c0ce604553/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c2b846b-5c8b-4982-abc9-809fdbffe82d/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/7c2b846b-5c8b-4982-abc9-809fdbffe82d/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c2b846b-5c8b-4982-abc9-809fdbffe82d/volumes/kubernetes.io~projected/kube-api-access-r4q2q:{mountpoint:/var/lib/kubelet/pods/7c2b846b-5c8b-4982-abc9-809fdbffe82d/volumes/kubernetes.io~projected/kube-api-access-r4q2q major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c2b846b-5c8b-4982-abc9-809fdbffe82d/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/7c2b846b-5c8b-4982-abc9-809fdbffe82d/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c2b846b-5c8b-4982-abc9-809fdbffe82d/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/7c2b846b-5c8b-4982-abc9-809fdbffe82d/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85784876-9397-4bec-8ade-878a15c09f40/volumes/kubernetes.io~projected/kube-api-access-4dzj7:{mountpoint:/var/lib/kubelet/pods/85784876-9397-4bec-8ade-878a15c09f40/volumes/kubernetes.io~projected/kube-api-access-4dzj7 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8618e1e8-4d71-4f7a-b2a1-1b950190f110/volumes/kubernetes.io~projected/kube-api-access-wjz2b:{mountpoint:/var/lib/kubelet/pods/8618e1e8-4d71-4f7a-b2a1-1b950190f110/volumes/kubernetes.io~projected/kube-api-access-wjz2b major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91cb4fd5-ed27-4a79-b172-be80b203bf0f/volumes/kubernetes.io~projected/kube-api-access-862mp:{mountpoint:/var/lib/kubelet/pods/91cb4fd5-ed27-4a79-b172-be80b203bf0f/volumes/kubernetes.io~projected/kube-api-access-862mp major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91cb4fd5-ed27-4a79-b172-be80b203bf0f/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/91cb4fd5-ed27-4a79-b172-be80b203bf0f/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b91ad19-9236-4395-9e54-09470929bf35/volumes/kubernetes.io~projected/kube-api-access-lqppx:{mountpoint:/var/lib/kubelet/pods/9b91ad19-9236-4395-9e54-09470929bf35/volumes/kubernetes.io~projected/kube-api-access-lqppx major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a128d9e6-7be3-4c73-88f3-ba6b9ea59971/volumes/kubernetes.io~projected/kube-api-access-j98n8:{mountpoint:/var/lib/kubelet/pods/a128d9e6-7be3-4c73-88f3-ba6b9ea59971/volumes/kubernetes.io~projected/kube-api-access-j98n8 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1ed8a22-43d0-4ca3-9f7b-900231ad3685/volumes/kubernetes.io~projected/kube-api-access-b4wlx:{mountpoint:/var/lib/kubelet/pods/a1ed8a22-43d0-4ca3-9f7b-900231ad3685/volumes/kubernetes.io~projected/kube-api-access-b4wlx major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b083c3f1-9c74-4313-a939-ff8aa5769c8d/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/b083c3f1-9c74-4313-a939-ff8aa5769c8d/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b083c3f1-9c74-4313-a939-ff8aa5769c8d/volumes/kubernetes.io~projected/kube-api-access-crplh:{mountpoint:/var/lib/kubelet/pods/b083c3f1-9c74-4313-a939-ff8aa5769c8d/volumes/kubernetes.io~projected/kube-api-access-crplh major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b083c3f1-9c74-4313-a939-ff8aa5769c8d/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/b083c3f1-9c74-4313-a939-ff8aa5769c8d/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b/volumes/kubernetes.io~projected/kube-api-access-44rfr:{mountpoint:/var/lib/kubelet/pods/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b/volumes/kubernetes.io~projected/kube-api-access-44rfr major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb2a916a-fc15-4bdc-b28a-c2473a543ce5/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/bb2a916a-fc15-4bdc-b28a-c2473a543ce5/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bdebb5b6-76a2-48c8-810d-6f80549a4f3c/volumes/kubernetes.io~projected/kube-api-access-kdnrv:{mountpoint:/var/lib/kubelet/pods/bdebb5b6-76a2-48c8-810d-6f80549a4f3c/volumes/kubernetes.io~projected/kube-api-access-kdnrv major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bdebb5b6-76a2-48c8-810d-6f80549a4f3c/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/bdebb5b6-76a2-48c8-810d-6f80549a4f3c/volumes/kubernetes.io~secret/default-certificate major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bdebb5b6-76a2-48c8-810d-6f80549a4f3c/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/bdebb5b6-76a2-48c8-810d-6f80549a4f3c/volumes/kubernetes.io~secret/stats-auth major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5c07387-ae45-48d3-8b9d-8f328bdfd63c/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/d5c07387-ae45-48d3-8b9d-8f328bdfd63c/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5c07387-ae45-48d3-8b9d-8f328bdfd63c/volumes/kubernetes.io~projected/kube-api-access-tnz7p:{mountpoint:/var/lib/kubelet/pods/d5c07387-ae45-48d3-8b9d-8f328bdfd63c/volumes/kubernetes.io~projected/kube-api-access-tnz7p major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5c07387-ae45-48d3-8b9d-8f328bdfd63c/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/d5c07387-ae45-48d3-8b9d-8f328bdfd63c/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5c07387-ae45-48d3-8b9d-8f328bdfd63c/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/d5c07387-ae45-48d3-8b9d-8f328bdfd63c/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d628518e-2ab7-4acb-acf7-6fc96f359a64/volumes/kubernetes.io~projected/kube-api-access-rh2r4:{mountpoint:/var/lib/kubelet/pods/d628518e-2ab7-4acb-acf7-6fc96f359a64/volumes/kubernetes.io~projected/kube-api-access-rh2r4 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699/volumes/kubernetes.io~projected/kube-api-access-rq7hj:{mountpoint:/var/lib/kubelet/pods/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699/volumes/kubernetes.io~projected/kube-api-access-rq7hj major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e534da40-0733-4670-ba7a-839e0e4b1b4f/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e534da40-0733-4670-ba7a-839e0e4b1b4f/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e534da40-0733-4670-ba7a-839e0e4b1b4f/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e534da40-0733-4670-ba7a-839e0e4b1b4f/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e534da40-0733-4670-ba7a-839e0e4b1b4f/volumes/kubernetes.io~projected/kube-api-access-5zjgc:{mountpoint:/var/lib/kubelet/pods/e534da40-0733-4670-ba7a-839e0e4b1b4f/volumes/kubernetes.io~projected/kube-api-access-5zjgc major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6c17513-0478-4a05-94dd-0cdad5facf06/volumes/kubernetes.io~projected/kube-api-access-rgsc7:{mountpoint:/var/lib/kubelet/pods/e6c17513-0478-4a05-94dd-0cdad5facf06/volumes/kubernetes.io~projected/kube-api-access-rgsc7 major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e79ec42c-3696-40e7-94de-57aad9a9f92c/volumes/kubernetes.io~projected/kube-api-access-4tghz:{mountpoint:/var/lib/kubelet/pods/e79ec42c-3696-40e7-94de-57aad9a9f92c/volumes/kubernetes.io~projected/kube-api-access-4tghz major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e79ec42c-3696-40e7-94de-57aad9a9f92c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e79ec42c-3696-40e7-94de-57aad9a9f92c/volumes/kubernetes.io~secret/cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb8ad2e6-7e4f-4c0a-9ea3-99ed98344ba6/volumes/kubernetes.io~projected/kube-api-access-xh798:{mountpoint:/var/lib/kubelet/pods/eb8ad2e6-7e4f-4c0a-9ea3-99ed98344ba6/volumes/kubernetes.io~projected/kube-api-access-xh798 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb932801-604f-430e-bfe3-8d4c9a1a7ca6/volumes/kubernetes.io~projected/kube-api-access-drbjx:{mountpoint:/var/lib/kubelet/pods/eb932801-604f-430e-bfe3-8d4c9a1a7ca6/volumes/kubernetes.io~projected/kube-api-access-drbjx major:0 minor:321 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6bbdceebc76c33be57bbd08d80d76ba7589cd4f502e2f5de8a0467da6f01943e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f8a97478b673b32b747fb5ee7c391f40435a7246b1390b5ec100d36352500030/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f0f7d8e81e5735a43c35c918241bba2f0fcee8921fdeb34040b7a2e35cd6d4fb/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/7f760eea9a3975c2ad336efa5dee9beaa77b67b3e93c4e134e3dc504ef6900ea/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/dac873aae8631c6eb19d094ec3ed589b18c91b662cd183bbc07fc6dc4b3acefb/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/6a36ec15aa1f91b7eadb1df048fd580c676cf08b20af4e7b19dc05065b6f4770/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/ecd1ceb40b79314d73147cca21acfb61a820f4f62b159675c949e8641bc5931d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e89e4fdd51579c33d78f197c7f88bc096ed61f767d477351da7f79c6ef5bf40d/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/4a45bf0aaede08a0d1660196fd3144085fea5238b5028265a7445c1862b44d5c/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/f1ae2feed23bcbc128ec46ba8b374077246096363ac4adfde9a5f0dd06bf4bb2/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/013d393c43fd899b6b917d5c4e16162138d0728941e36f2e2e7daf6752de4010/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/1e5e19440b57e288069370a4b048cf3dfa21e24d989c9f9414d1784fa51c341b/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/0dcca86aaf3b3d5c12b988555c9ceaa6a63b76c60373203ed6a7102b1d3c3eed/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/454f6661875e44f826630d0e768e9d782e4d7d14e2f4c33074638a6f1970c4ed/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/cdc14a7f327e89caa72a43d8b026ddc9429575facd565a63a113c17d75c52ba3/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/8464524039e08aa5c4eb436b621fa85b0879a5352cc498416c47e4139b89f698/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/929c36ce8aecd0bc4f8e584c0fb0207e2fa5df4d6ca0b7cf87fb6d885bd11753/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/282303b8827a05bd797a0713f3907f5c459f714d1339b4d15b69091c57b7c47c/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/dc61998269925ea6a6f8913d23a89f8801d505ca7c53b5f3eb8e9dd8223b4396/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/a6c2f7cbf446730b8ef7b819b15c53011c67930ba6f3d81075271a0ba1d6beb3/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7f45f1065839869e3838a0503bb16bec1f16b51783440e3d261a710df421c3e2/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/653535024faabff1ee3fa381305a00f90b8f8dbdef7c5b176096faed93fc1d0f/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/38754c1ed432b899329f827c24840babb13680451d822dbe716b703e14f3fe38/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/a43f90b5559209e645a51b8d2f0e46287605dc44bc353915260d421c3e13c73a/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/5708265a87307d850b094f72d237ad9306939a360c29ec30bb33973e3afc6982/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/71ab670a7515921ca3b666443436557450e45ca1c70c3d43f79d389350f0133d/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/d43e9502043c3eaa6bac7426847e7ef69e12b94778432f0b47b32573509d3a9f/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/6855e7baf375941b1c428890083bc5300d111a1d7f895f9ee385e1d78a2459d8/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/2087786604a4cd22eb1e951e2cccdeeff12c7b1883ddfca3a9e55e96b72d58a9/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/be6b851838f44b7e5848aa330550ba83de9b8b5c2ad047390cbacd169ca23009/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/db1d6b4b332af6ce512a5298ba70ea060316cc6e5356e6eef9efa5ce3b83af73/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/c507f1fa1a2a0586167bb39301ab262fa67e32e133474941a13f3d996d7a420d/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/9e8a7c7f0b4578fdeaae05061288348136b897eec492ccddb0a98074c4cc9455/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/d656bc75251c315510c49c1705ac858f03640cae686dc9814486dadd7b91bc43/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/358a58b0c5386b3c30ef147115b9667b5002d833756e27d443403d6cc44ae01a/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/6d89c117db3dad0e7dadf24b4d3fcfb6bfc3ee35dd2b96184bc9f52905c24a79/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/78d9705bc029ddcd9b07fafda34ca1d006ff1849ce76f640063d07a9157c6cde/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/1060ac331e37c659a3b9513faf6379b2c064ec3081c31044020949cdf154790e/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/69d72b3e595f6fe1f73f516fd03e2870c36debda9d87f99cb4a1394ad24e7b2d/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/3d7b2ef578437cf52c51bae2503d6f6523a2b0736ce65f9834e9391c9570af27/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/49e964b547ffd4a0a304d3811a0c3f7693f5d0dd9496e3603c7dd740a6c6ad0a/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/1739b5eeefd79fe04880a680e654a79c71624d0470611b6d3ad0274792d4cb29/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/e03f4ce2e3e24c9683fbf982c3b9ad6e7f298765bd5aef29f45813b6eaf95d26/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/268716b516f81ae59bb5f6d555f8d8a3074d50161e2df8aadfa4a2dbc999ecfb/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/21f0036bddc2950373343f04c5b40dcc63bb87192758e44c5f7f557d831ef992/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/a2e851480cc029f9eb94286ea7f3b8b07be67d73ffbc244d8156665de8d1489e/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/fa9bfa2883b367efb0b60d13f3d47290a2132b00b0de82e20ab9cef8566179b5/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/f03417137de95f6af7d1cebc0270529e70882c31b5875f7156ac495ecad53abb/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/af7e226cde8bcc8fe3b96c738ba15c8afefbbe7584a4261f88da4155912d0955/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/046321c31008f670f6117a4c503ffb0dc4445aae4b636c190d054fc1f04e3671/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/e3a171f989eb59fcd549cbbfe7197e5d5c8c25a0e453d8fb507c8079c33cc8f3/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/1b6d86753c7e979d72a441b8a271e87d4b7f54052ac98a35f634cba14b4da8ce/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/fd33058f8ea0600595eb76ea7db37608b31475adfdb779d7c04a1190143e6f2c/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/017a8bf3b3bb049c511fa0b26dce1f711406518593764eba892ea2a65cd98115/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/2bb7d5d49a66bb0a8e2f88621216f2100cf1ee52633f6cf36b17d02cec705a33/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/286155eadc822675eb40708366ca4b8fa0cc7b697a0e9a4f40f15ce59fae73ad/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/828324f2473569494db90eac358349f616cb9979a6b5a7323a0dd63719684bfb/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/bb0602957d1256329e3e44de9a231232958390d3b6a6926579cad97d32859e99/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/641bf0b712a7bc052179786f8dafec98f2d64eb56c380d448f909661b31e6f84/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/97aa5880c17925948121e8364aa2d1eefcc0f66358479c55383a9b3ac89a9f79/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 01:24:43.390250 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.388543 5044 manager.go:217] Machine: {Timestamp:2026-11-02 01:24:43.388091411 +0000 UTC m=+0.059344085 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3055845 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29d8484f17f5e354463354903f2582 SystemUUID:ec29d848-4f17-f5e3-5446-3354903f2582 BootID:40e3b1ce-4f70-40d8-9354-27758804a79e Filesystems:[{Device:/var/lib/kubelet/pods/e79ec42c-3696-40e7-94de-57aad9a9f92c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60ee38920eb119bf15b637d23b79c11689392dd6adbec04062c5121447e03e03/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/194ec6187c7b572797fd8f78e02cb66a027fb1b3f1f870aa4adda003f033d9e7/userdata/shm DeviceMajor:0 DeviceMinor:322 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9fcaa49884d943ea43a86669a9512e72925c2d85280c94b971f08c50e553d79f/userdata/shm DeviceMajor:0 DeviceMinor:339 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a1ed8a22-43d0-4ca3-9f7b-900231ad3685/volumes/kubernetes.io~projected/kube-api-access-b4wlx DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f28823619809a7408c348ae1039f3bde4f82f41d60899137778532d11d51718c/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bdebb5b6-76a2-48c8-810d-6f80549a4f3c/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8618e1e8-4d71-4f7a-b2a1-1b950190f110/volumes/kubernetes.io~projected/kube-api-access-wjz2b DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2e031e5c-cb8a-42b9-b334-1cbe61a9c59f/volumes/kubernetes.io~projected/kube-api-access-6z8mm DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eb8ad2e6-7e4f-4c0a-9ea3-99ed98344ba6/volumes/kubernetes.io~projected/kube-api-access-xh798 DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e70bc567d899951d4b95692ad723aab0eee4d2d732c291c0b99e2ae4a2fd024a/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c2b846b-5c8b-4982-abc9-809fdbffe82d/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c285da3-2e11-498d-94f1-76c0ce604553/volumes/kubernetes.io~projected/kube-api-access-v8wfd DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99/volumes/kubernetes.io~projected/kube-api-access-jg9vq DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e6c17513-0478-4a05-94dd-0cdad5facf06/volumes/kubernetes.io~projected/kube-api-access-rgsc7 DeviceMajor:0 DeviceMinor:326 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/671ea2856c9b53a7db137f865f26d6585cd38fb6098f982fdcc8526857b37f9f/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b136b6d11358e2172ebba343dc7590986e27110dc3800b9d61c9491d619191a5/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ffc3d4e4c98fde42495e9d0bf5b1cc1a3a0e1b490ddd8ab72f086b1dd7fe9787/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb932801-604f-430e-bfe3-8d4c9a1a7ca6/volumes/kubernetes.io~projected/kube-api-access-drbjx DeviceMajor:0 DeviceMinor:321 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aef0f7510a57f5b33b937b4d573d2417f0dbf9f2f55ee57de1f762e2ea09b55d/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1b449089fc1b096507ccc81210b0ab22d0eef7f4816aa2fdafe9b3542bdbd4a/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c285da3-2e11-498d-94f1-76c0ce604553/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/780e6524fb0d75e94a6e9d5e7000be249c6dc9f4a2fdae31a49b93c800814f62/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/363dffa0-70cf-4fe0-996f-3c4d2f3b1a50/volumes/kubernetes.io~projected/kube-api-access-bxmrq DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4abd5311-4a92-4ac5-8aa0-779dd8774577/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e534da40-0733-4670-ba7a-839e0e4b1b4f/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b91ad19-9236-4395-9e54-09470929bf35/volumes/kubernetes.io~projected/kube-api-access-lqppx DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a128d9e6-7be3-4c73-88f3-ba6b9ea59971/volumes/kubernetes.io~projected/kube-api-access-j98n8 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbbaa485276e9e80414df1c76559d8406a6ebf38307eccac18eda98c904bffde/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91cb4fd5-ed27-4a79-b172-be80b203bf0f/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/028b56de-c4b9-47c1-8bfa-761877176429/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:356 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e534da40-0733-4670-ba7a-839e0e4b1b4f/volumes/kubernetes.io~projected/kube-api-access-5zjgc DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/624e8843c916c2fcc231815c561ff5315e6251132d93db11dbffb2278bb17833/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54db4b99-9007-429a-a267-b4924a8729e6/volumes/kubernetes.io~projected/kube-api-access-57blk DeviceMajor:0 DeviceMinor:316 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c2b846b-5c8b-4982-abc9-809fdbffe82d/volumes/kubernetes.io~projected/kube-api-access-r4q2q DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/91cb4fd5-ed27-4a79-b172-be80b203bf0f/volumes/kubernetes.io~projected/kube-api-access-862mp DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/270cd97377d77588e4c335e61b9b1eba3f5ea162f96bc871d31f4f2e1213cc10/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5c07387-ae45-48d3-8b9d-8f328bdfd63c/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f2e12d4751d7b88c57875fc67990526ffd54c6b50370cda54f8241789bdf974/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/25437e9b-ca22-4103-ac9e-c5823f2bbbaf/volumes/kubernetes.io~projected/kube-api-access-724nj DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d5c07387-ae45-48d3-8b9d-8f328bdfd63c/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/79f97db3-814b-4849-ab8d-305dbcd3449c/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e79ec42c-3696-40e7-94de-57aad9a9f92c/volumes/kubernetes.io~projected/kube-api-access-4tghz DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/35ae4588-f166-4563-aae7-b34b5b053d0f/volumes/kubernetes.io~projected/kube-api-access-6s6ct DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2c8cbfce-aaf5-4362-930e-b15539b2dbc4/volumes/kubernetes.io~projected/kube-api-access-jvshf DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bdebb5b6-76a2-48c8-810d-6f80549a4f3c/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bdebb5b6-76a2-48c8-810d-6f80549a4f3c/volumes/kubernetes.io~projected/kube-api-access-kdnrv DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c4476138a7bf2989aac4600b621afb29324711330f782eb69f9bbce13d98075/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b083c3f1-9c74-4313-a939-ff8aa5769c8d/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb877fe27b5a235f783aeb6c121b0d7df45af00d84d72cacb0272dec49d6fd58/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/411ae169b7770debda9684191543d54f36415adc6398062ead31c1100c6c1d81/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d2fee6e42afcc68e2cbe0d970b893f2203dbf493985ccb97070845466de3e72/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4abd5311-4a92-4ac5-8aa0-779dd8774577/volumes/kubernetes.io~projected/kube-api-access-tpwzf DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c2b846b-5c8b-4982-abc9-809fdbffe82d/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c2b846b-5c8b-4982-abc9-809fdbffe82d/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85784876-9397-4bec-8ade-878a15c09f40/volumes/kubernetes.io~projected/kube-api-access-4dzj7 DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e417561d9def4122cb070f5244c8735d39aa9d61f3ac1d114ff9a4e5e5c4a4f/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cae0ce038d06591c037675c225f487723b7f02925e0eddf2839d06909619ccbb/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d5c07387-ae45-48d3-8b9d-8f328bdfd63c/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bb2a916a-fc15-4bdc-b28a-c2473a543ce5/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d604ba2867e44ccc9b62c8fe569b770ac1a144bca0d2ddfedade3e526a2ff0b/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b45cf67f3d40c4644ea1e2a59eeb281e816e65de1f906142311859c233cb806/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35ae4588-f166-4563-aae7-b34b5b053d0f/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b/volumes/kubernetes.io~projected/kube-api-access-44rfr DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/673cb927-a5fc-4237-9b4c-fb196801d70c/volumes/kubernetes.io~projected/kube-api-access-xrnrh DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1fbaf835ff8f3a7565f0d4aefabd3fbdfac25aa4676e785fad3e60cc44e6a38/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/028b56de-c4b9-47c1-8bfa-761877176429/volumes/kubernetes.io~projected/kube-api-access-k4h6s DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a6d9848ff171086e151f397491a8f514dc8832592a239c33122c3284875456f/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e534da40-0733-4670-ba7a-839e0e4b1b4f/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b083c3f1-9c74-4313-a939-ff8aa5769c8d/volumes/kubernetes.io~projected/kube-api-access-crplh DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b083c3f1-9c74-4313-a939-ff8aa5769c8d/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/476d365d-dd0a-41e6-a186-5e92551be763/volumes/kubernetes.io~projected/kube-api-access-dc4n8 DeviceMajor:0 DeviceMinor:331 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d628518e-2ab7-4acb-acf7-6fc96f359a64/volumes/kubernetes.io~projected/kube-api-access-rh2r4 DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5c07387-ae45-48d3-8b9d-8f328bdfd63c/volumes/kubernetes.io~projected/kube-api-access-tnz7p DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699/volumes/kubernetes.io~projected/kube-api-access-rq7hj DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/59ccadf3-202d-481d-8c8b-c4fa86c92b8b/volumes/kubernetes.io~projected/kube-api-access-xp982 DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06f2e60e44f6b1ae0d0ce0b7cdac42b43ef1a76ff92b2a849246f46e15896d45/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:06f2e60e44f6b1a MacAddress:42:69:de:b7:13:0a Speed:10000 Mtu:8901} {Name:0f2e12d4751d7b8 MacAddress:3a:55:5b:ec:b1:ff Speed:10000 Mtu:8901} {Name:194ec6187c7b572 MacAddress:06:08:de:ee:5a:ad Speed:10000 Mtu:8901} {Name:270cd97377d7758 MacAddress:ca:79:5b:6a:7a:f7 Speed:10000 Mtu:8901} {Name:2c4476138a7bf29 MacAddress:1a:4d:39:bb:d4:f6 Speed:10000 Mtu:8901} {Name:411ae169b7770de MacAddress:fa:08:96:72:37:55 Speed:10000 Mtu:8901} {Name:4e417561d9def41 MacAddress:0a:ae:30:97:b0:33 Speed:10000 Mtu:8901} {Name:780e6524fb0d75e MacAddress:ae:94:81:7a:2f:df Speed:10000 Mtu:8901} {Name:8d2fee6e42afcc6 MacAddress:92:4b:ab:f9:cf:5d Speed:10000 Mtu:8901} {Name:9fcaa49884d943e MacAddress:d6:9b:7b:17:21:2e Speed:10000 Mtu:8901} {Name:aef0f7510a57f5b MacAddress:12:64:a1:30:2d:24 Speed:10000 Mtu:8901} {Name:b136b6d11358e21 MacAddress:fa:b7:2c:1d:e1:99 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:64:d9:a7:c9:d1 Speed:0 Mtu:9001} {Name:br-int MacAddress:ee:07:c0:f8:f0:c5 Speed:0 Mtu:8901} {Name:d1b449089fc1b09 MacAddress:ea:8b:14:1e:d1:21 Speed:10000 Mtu:8901} {Name:e1fbaf835ff8f3a MacAddress:4e:2f:7b:0b:0b:82 Speed:10000 Mtu:8901} {Name:e70bc567d899951 MacAddress:2e:a9:0c:13:a3:8c Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:64:d9:a7:c9:d1 Speed:0 Mtu:9001} {Name:ffc3d4e4c98fde4 MacAddress:f2:22:22:f6:b6:3e Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:c6:14:86:75:43:45 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4a:31:7c:01:c8:4f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 01:24:43.390250 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.388910 5044 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 01:24:43.390250 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.388953 5044 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 01:24:43.390250 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.389257 5044 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 01:24:43.390250 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.389285 5044 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-6.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 01:24:43.390250 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.389402 5044 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 01:24:43.390250 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.389410 5044 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 01:24:43.390250 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.389420 5044 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 01:24:43.390250 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.389439 5044 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 01:24:43.390250 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.389457 5044 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 01:24:43.390250 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.389603 5044 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 01:24:43.390250 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.389730 5044 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 01:24:43.390250 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.389775 5044 kubelet.go:493] "Attempting to sync node with API server" Nov 02 01:24:43.390250 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.389789 5044 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 01:24:43.390250 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.389802 5044 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 01:24:43.390250 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.389811 5044 kubelet.go:405] "Adding apiserver pod source" Nov 02 01:24:43.390250 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.389819 5044 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 01:24:43.391019 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.390651 5044 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 01:24:43.391019 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.390662 5044 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 01:24:43.391197 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.391182 5044 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 01:24:43.404100 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.404083 5044 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 01:24:43.404100 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.404099 5044 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 01:24:43.404197 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.404186 5044 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 01:24:43.404364 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.404353 5044 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 01:24:43.404395 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.404382 5044 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 01:24:43.404475 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.404467 5044 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 01:24:43.404503 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.404481 5044 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 01:24:43.404503 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.404488 5044 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 01:24:43.404503 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.404493 5044 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 01:24:43.404503 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.404499 5044 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 01:24:43.404503 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.404505 5044 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 01:24:43.404647 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.404511 5044 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 01:24:43.404647 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.404516 5044 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 01:24:43.404647 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.404522 5044 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 01:24:43.404647 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.404528 5044 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 01:24:43.404647 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.404540 5044 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 01:24:43.404647 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.404550 5044 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 01:24:43.404647 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.404582 5044 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 01:24:43.404647 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.404588 5044 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 01:24:43.407918 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.407904 5044 server.go:1263] "Started kubelet" Nov 02 01:24:43.408026 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.407958 5044 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 01:24:43.408160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.408094 5044 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 01:24:43.408242 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.408183 5044 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 01:24:43.408506 ip-10-0-129-6 systemd[1]: Started Kubernetes Kubelet. Nov 02 01:24:43.408919 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.408704 5044 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 01:24:43.409616 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.409594 5044 server.go:317] "Adding debug handlers to kubelet server" Nov 02 01:24:43.415757 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.415559 5044 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 01:24:43.415757 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.415600 5044 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 01:24:43.415757 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.415706 5044 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 01:19:02 +0000 UTC" deadline="2028-04-08 17:01:04.032528358 +0000 UTC" Nov 02 01:24:43.415757 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.415722 5044 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 01:24:43.415757 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.415731 5044 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12567h36m20.616801181s" Nov 02 01:24:43.415757 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.415738 5044 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 01:24:43.416080 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.415805 5044 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 01:24:43.416080 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:43.415964 5044 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-6.us-west-1.compute.internal\" not found" Nov 02 01:24:43.416752 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.416737 5044 factory.go:55] Registering systemd factory Nov 02 01:24:43.416752 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.416754 5044 factory.go:223] Registration of the systemd container factory successfully Nov 02 01:24:43.417129 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.417094 5044 factory.go:153] Registering CRI-O factory Nov 02 01:24:43.417129 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.417116 5044 factory.go:223] Registration of the crio container factory successfully Nov 02 01:24:43.417269 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.417171 5044 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 01:24:43.417269 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.417192 5044 factory.go:103] Registering Raw factory Nov 02 01:24:43.417269 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.417209 5044 manager.go:1199] Started watching for new ooms in manager Nov 02 01:24:43.417560 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.417546 5044 manager.go:319] Starting recovery of all containers Nov 02 01:24:43.418945 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.418915 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="028b56de-c4b9-47c1-8bfa-761877176429" volumeName="kubernetes.io/projected/028b56de-c4b9-47c1-8bfa-761877176429-kube-api-access-k4h6s" seLinuxMountContext="" Nov 02 01:24:43.418987 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.418947 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e031e5c-cb8a-42b9-b334-1cbe61a9c59f" volumeName="kubernetes.io/projected/2e031e5c-cb8a-42b9-b334-1cbe61a9c59f-kube-api-access-6z8mm" seLinuxMountContext="" Nov 02 01:24:43.418987 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.418956 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79f97db3-814b-4849-ab8d-305dbcd3449c" volumeName="kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret" seLinuxMountContext="" Nov 02 01:24:43.418987 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.418964 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c285da3-2e11-498d-94f1-76c0ce604553" volumeName="kubernetes.io/configmap/7c285da3-2e11-498d-94f1-76c0ce604553-trusted-ca" seLinuxMountContext="" Nov 02 01:24:43.418987 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.418971 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b083c3f1-9c74-4313-a939-ff8aa5769c8d" volumeName="kubernetes.io/projected/b083c3f1-9c74-4313-a939-ff8aa5769c8d-kube-api-access-crplh" seLinuxMountContext="" Nov 02 01:24:43.418987 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.418978 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdebb5b6-76a2-48c8-810d-6f80549a4f3c" volumeName="kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-default-certificate" seLinuxMountContext="" Nov 02 01:24:43.418987 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.418985 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5a0b9c-57f5-4741-94f3-fb9fe7b7e699" volumeName="kubernetes.io/projected/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-kube-api-access-rq7hj" seLinuxMountContext="" Nov 02 01:24:43.419160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.418994 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb932801-604f-430e-bfe3-8d4c9a1a7ca6" volumeName="kubernetes.io/projected/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-kube-api-access-drbjx" seLinuxMountContext="" Nov 02 01:24:43.419160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419006 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59ccadf3-202d-481d-8c8b-c4fa86c92b8b" volumeName="kubernetes.io/projected/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-kube-api-access-xp982" seLinuxMountContext="" Nov 02 01:24:43.419160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419014 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91cb4fd5-ed27-4a79-b172-be80b203bf0f" volumeName="kubernetes.io/empty-dir/91cb4fd5-ed27-4a79-b172-be80b203bf0f-snapshots" seLinuxMountContext="" Nov 02 01:24:43.419160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419021 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b91ad19-9236-4395-9e54-09470929bf35" volumeName="kubernetes.io/configmap/9b91ad19-9236-4395-9e54-09470929bf35-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 01:24:43.419160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419028 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb2a916a-fc15-4bdc-b28a-c2473a543ce5" volumeName="kubernetes.io/configmap/bb2a916a-fc15-4bdc-b28a-c2473a543ce5-konnectivity-ca" seLinuxMountContext="" Nov 02 01:24:43.419160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419036 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5a0b9c-57f5-4741-94f3-fb9fe7b7e699" volumeName="kubernetes.io/configmap/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-config-volume" seLinuxMountContext="" Nov 02 01:24:43.419160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419042 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e79ec42c-3696-40e7-94de-57aad9a9f92c" volumeName="kubernetes.io/secret/e79ec42c-3696-40e7-94de-57aad9a9f92c-cert" seLinuxMountContext="" Nov 02 01:24:43.419160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419051 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41" volumeName="kubernetes.io/configmap/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-nginx-conf" seLinuxMountContext="" Nov 02 01:24:43.419160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419058 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4abd5311-4a92-4ac5-8aa0-779dd8774577" volumeName="kubernetes.io/secret/4abd5311-4a92-4ac5-8aa0-779dd8774577-serving-cert" seLinuxMountContext="" Nov 02 01:24:43.419160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419065 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54db4b99-9007-429a-a267-b4924a8729e6" volumeName="kubernetes.io/projected/54db4b99-9007-429a-a267-b4924a8729e6-kube-api-access-57blk" seLinuxMountContext="" Nov 02 01:24:43.419160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419077 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99" volumeName="kubernetes.io/projected/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-kube-api-access-jg9vq" seLinuxMountContext="" Nov 02 01:24:43.419160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419087 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91cb4fd5-ed27-4a79-b172-be80b203bf0f" volumeName="kubernetes.io/empty-dir/91cb4fd5-ed27-4a79-b172-be80b203bf0f-tmp" seLinuxMountContext="" Nov 02 01:24:43.419160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419094 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb2a916a-fc15-4bdc-b28a-c2473a543ce5" volumeName="kubernetes.io/secret/bb2a916a-fc15-4bdc-b28a-c2473a543ce5-agent-certs" seLinuxMountContext="" Nov 02 01:24:43.419160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419101 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85784876-9397-4bec-8ade-878a15c09f40" volumeName="kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7" seLinuxMountContext="" Nov 02 01:24:43.419160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419108 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5c07387-ae45-48d3-8b9d-8f328bdfd63c" volumeName="kubernetes.io/configmap/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-trusted-ca" seLinuxMountContext="" Nov 02 01:24:43.419160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419114 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5c07387-ae45-48d3-8b9d-8f328bdfd63c" volumeName="kubernetes.io/secret/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-image-registry-private-configuration" seLinuxMountContext="" Nov 02 01:24:43.419160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419135 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5c07387-ae45-48d3-8b9d-8f328bdfd63c" volumeName="kubernetes.io/secret/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-installation-pull-secrets" seLinuxMountContext="" Nov 02 01:24:43.419160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419146 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25437e9b-ca22-4103-ac9e-c5823f2bbbaf" volumeName="kubernetes.io/projected/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-kube-api-access-724nj" seLinuxMountContext="" Nov 02 01:24:43.419160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419158 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="476d365d-dd0a-41e6-a186-5e92551be763" volumeName="kubernetes.io/projected/476d365d-dd0a-41e6-a186-5e92551be763-kube-api-access-dc4n8" seLinuxMountContext="" Nov 02 01:24:43.419160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419167 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4abd5311-4a92-4ac5-8aa0-779dd8774577" volumeName="kubernetes.io/configmap/4abd5311-4a92-4ac5-8aa0-779dd8774577-config" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419176 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54db4b99-9007-429a-a267-b4924a8729e6" volumeName="kubernetes.io/empty-dir/54db4b99-9007-429a-a267-b4924a8729e6-data-volume" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419196 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c2b846b-5c8b-4982-abc9-809fdbffe82d" volumeName="kubernetes.io/configmap/7c2b846b-5c8b-4982-abc9-809fdbffe82d-trusted-ca" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419204 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb932801-604f-430e-bfe3-8d4c9a1a7ca6" volumeName="kubernetes.io/configmap/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-service-ca-bundle" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419212 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35ae4588-f166-4563-aae7-b34b5b053d0f" volumeName="kubernetes.io/projected/35ae4588-f166-4563-aae7-b34b5b053d0f-kube-api-access-6s6ct" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419220 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35ae4588-f166-4563-aae7-b34b5b053d0f" volumeName="kubernetes.io/secret/35ae4588-f166-4563-aae7-b34b5b053d0f-serving-cert" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419235 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="673cb927-a5fc-4237-9b4c-fb196801d70c" volumeName="kubernetes.io/projected/673cb927-a5fc-4237-9b4c-fb196801d70c-kube-api-access-xrnrh" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419243 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c285da3-2e11-498d-94f1-76c0ce604553" volumeName="kubernetes.io/secret/7c285da3-2e11-498d-94f1-76c0ce604553-serving-cert" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419250 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91cb4fd5-ed27-4a79-b172-be80b203bf0f" volumeName="kubernetes.io/secret/91cb4fd5-ed27-4a79-b172-be80b203bf0f-serving-cert" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419258 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1ed8a22-43d0-4ca3-9f7b-900231ad3685" volumeName="kubernetes.io/empty-dir/a1ed8a22-43d0-4ca3-9f7b-900231ad3685-tmp-dir" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419266 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6c17513-0478-4a05-94dd-0cdad5facf06" volumeName="kubernetes.io/projected/e6c17513-0478-4a05-94dd-0cdad5facf06-kube-api-access-rgsc7" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419273 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8618e1e8-4d71-4f7a-b2a1-1b950190f110" volumeName="kubernetes.io/projected/8618e1e8-4d71-4f7a-b2a1-1b950190f110-kube-api-access-wjz2b" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419281 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91cb4fd5-ed27-4a79-b172-be80b203bf0f" volumeName="kubernetes.io/configmap/91cb4fd5-ed27-4a79-b172-be80b203bf0f-trusted-ca-bundle" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419288 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b083c3f1-9c74-4313-a939-ff8aa5769c8d" volumeName="kubernetes.io/configmap/b083c3f1-9c74-4313-a939-ff8aa5769c8d-env-overrides" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419320 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5c07387-ae45-48d3-8b9d-8f328bdfd63c" volumeName="kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-bound-sa-token" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419331 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb8ad2e6-7e4f-4c0a-9ea3-99ed98344ba6" volumeName="kubernetes.io/projected/eb8ad2e6-7e4f-4c0a-9ea3-99ed98344ba6-kube-api-access-xh798" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419343 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35ae4588-f166-4563-aae7-b34b5b053d0f" volumeName="kubernetes.io/configmap/35ae4588-f166-4563-aae7-b34b5b053d0f-config" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419352 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b91ad19-9236-4395-9e54-09470929bf35" volumeName="kubernetes.io/configmap/9b91ad19-9236-4395-9e54-09470929bf35-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419360 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1ed8a22-43d0-4ca3-9f7b-900231ad3685" volumeName="kubernetes.io/projected/a1ed8a22-43d0-4ca3-9f7b-900231ad3685-kube-api-access-b4wlx" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419367 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdebb5b6-76a2-48c8-810d-6f80549a4f3c" volumeName="kubernetes.io/projected/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-kube-api-access-kdnrv" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419375 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da5a0b9c-57f5-4741-94f3-fb9fe7b7e699" volumeName="kubernetes.io/empty-dir/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-tmp-dir" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419386 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e534da40-0733-4670-ba7a-839e0e4b1b4f" volumeName="kubernetes.io/empty-dir/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-tuned" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419397 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c285da3-2e11-498d-94f1-76c0ce604553" volumeName="kubernetes.io/configmap/7c285da3-2e11-498d-94f1-76c0ce604553-config" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419405 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59ccadf3-202d-481d-8c8b-c4fa86c92b8b" volumeName="kubernetes.io/configmap/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-telemetry-config" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419414 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99" volumeName="kubernetes.io/configmap/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-cni-binary-copy" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419421 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c2b846b-5c8b-4982-abc9-809fdbffe82d" volumeName="kubernetes.io/secret/7c2b846b-5c8b-4982-abc9-809fdbffe82d-installation-pull-secrets" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419429 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b" volumeName="kubernetes.io/configmap/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-config" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419438 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5c07387-ae45-48d3-8b9d-8f328bdfd63c" volumeName="kubernetes.io/configmap/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-certificates" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419445 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb932801-604f-430e-bfe3-8d4c9a1a7ca6" volumeName="kubernetes.io/empty-dir/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-archives-path" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419453 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b91ad19-9236-4395-9e54-09470929bf35" volumeName="kubernetes.io/configmap/9b91ad19-9236-4395-9e54-09470929bf35-cni-binary-copy" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419472 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d628518e-2ab7-4acb-acf7-6fc96f359a64" volumeName="kubernetes.io/projected/d628518e-2ab7-4acb-acf7-6fc96f359a64-kube-api-access-rh2r4" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419482 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e534da40-0733-4670-ba7a-839e0e4b1b4f" volumeName="kubernetes.io/empty-dir/e534da40-0733-4670-ba7a-839e0e4b1b4f-tmp" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419492 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e79ec42c-3696-40e7-94de-57aad9a9f92c" volumeName="kubernetes.io/projected/e79ec42c-3696-40e7-94de-57aad9a9f92c-kube-api-access-4tghz" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419500 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="028b56de-c4b9-47c1-8bfa-761877176429" volumeName="kubernetes.io/secret/028b56de-c4b9-47c1-8bfa-761877176429-signing-key" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419509 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c8cbfce-aaf5-4362-930e-b15539b2dbc4" volumeName="kubernetes.io/projected/2c8cbfce-aaf5-4362-930e-b15539b2dbc4-kube-api-access-jvshf" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419516 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="363dffa0-70cf-4fe0-996f-3c4d2f3b1a50" volumeName="kubernetes.io/projected/363dffa0-70cf-4fe0-996f-3c4d2f3b1a50-kube-api-access-bxmrq" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419524 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99" volumeName="kubernetes.io/configmap/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-daemon-config" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419531 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c2b846b-5c8b-4982-abc9-809fdbffe82d" volumeName="kubernetes.io/empty-dir/7c2b846b-5c8b-4982-abc9-809fdbffe82d-ca-trust-extracted" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419539 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c2b846b-5c8b-4982-abc9-809fdbffe82d" volumeName="kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-kube-api-access-r4q2q" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419551 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c2b846b-5c8b-4982-abc9-809fdbffe82d" volumeName="kubernetes.io/secret/7c2b846b-5c8b-4982-abc9-809fdbffe82d-image-registry-private-configuration" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419561 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91cb4fd5-ed27-4a79-b172-be80b203bf0f" volumeName="kubernetes.io/projected/91cb4fd5-ed27-4a79-b172-be80b203bf0f-kube-api-access-862mp" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419568 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="028b56de-c4b9-47c1-8bfa-761877176429" volumeName="kubernetes.io/configmap/028b56de-c4b9-47c1-8bfa-761877176429-signing-cabundle" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419575 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c2b846b-5c8b-4982-abc9-809fdbffe82d" volumeName="kubernetes.io/configmap/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-certificates" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419583 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c2b846b-5c8b-4982-abc9-809fdbffe82d" volumeName="kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-bound-sa-token" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419593 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b083c3f1-9c74-4313-a939-ff8aa5769c8d" volumeName="kubernetes.io/configmap/b083c3f1-9c74-4313-a939-ff8aa5769c8d-ovnkube-config" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419606 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b083c3f1-9c74-4313-a939-ff8aa5769c8d" volumeName="kubernetes.io/configmap/b083c3f1-9c74-4313-a939-ff8aa5769c8d-ovnkube-script-lib" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419617 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b" volumeName="kubernetes.io/projected/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-kube-api-access-44rfr" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419629 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c285da3-2e11-498d-94f1-76c0ce604553" volumeName="kubernetes.io/projected/7c285da3-2e11-498d-94f1-76c0ce604553-kube-api-access-v8wfd" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419640 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4abd5311-4a92-4ac5-8aa0-779dd8774577" volumeName="kubernetes.io/projected/4abd5311-4a92-4ac5-8aa0-779dd8774577-kube-api-access-tpwzf" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419652 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91cb4fd5-ed27-4a79-b172-be80b203bf0f" volumeName="kubernetes.io/configmap/91cb4fd5-ed27-4a79-b172-be80b203bf0f-service-ca-bundle" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419661 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a128d9e6-7be3-4c73-88f3-ba6b9ea59971" volumeName="kubernetes.io/projected/a128d9e6-7be3-4c73-88f3-ba6b9ea59971-kube-api-access-j98n8" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419685 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5c07387-ae45-48d3-8b9d-8f328bdfd63c" volumeName="kubernetes.io/empty-dir/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-ca-trust-extracted" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419694 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5c07387-ae45-48d3-8b9d-8f328bdfd63c" volumeName="kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-kube-api-access-tnz7p" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419701 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b91ad19-9236-4395-9e54-09470929bf35" volumeName="kubernetes.io/projected/9b91ad19-9236-4395-9e54-09470929bf35-kube-api-access-lqppx" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419709 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b083c3f1-9c74-4313-a939-ff8aa5769c8d" volumeName="kubernetes.io/secret/b083c3f1-9c74-4313-a939-ff8aa5769c8d-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419717 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb932801-604f-430e-bfe3-8d4c9a1a7ca6" volumeName="kubernetes.io/configmap/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-trusted-ca-bundle" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419725 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="673cb927-a5fc-4237-9b4c-fb196801d70c" volumeName="kubernetes.io/configmap/673cb927-a5fc-4237-9b4c-fb196801d70c-iptables-alerter-script" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419733 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a128d9e6-7be3-4c73-88f3-ba6b9ea59971" volumeName="kubernetes.io/configmap/a128d9e6-7be3-4c73-88f3-ba6b9ea59971-serviceca" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419741 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bdebb5b6-76a2-48c8-810d-6f80549a4f3c" volumeName="kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-stats-auth" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419749 5044 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e534da40-0733-4670-ba7a-839e0e4b1b4f" volumeName="kubernetes.io/projected/e534da40-0733-4670-ba7a-839e0e4b1b4f-kube-api-access-5zjgc" seLinuxMountContext="" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419761 5044 reconstruct.go:97] "Volume reconstruction finished" Nov 02 01:24:43.419795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.419768 5044 reconciler.go:29] "Reconciler: start to sync state" Nov 02 01:24:43.422861 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.422162 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/1.log" Nov 02 01:24:43.429022 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.429000 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/1.log" Nov 02 01:24:43.436006 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.435972 5044 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 01:24:43.443991 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.443973 5044 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 01:24:43.443991 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.443991 5044 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 01:24:43.444122 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.444010 5044 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 01:24:43.444122 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:43.444045 5044 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 01:24:43.461245 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.461225 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/1.log" Nov 02 01:24:43.461356 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.461250 5044 generic.go:356] "Generic (PLEG): container finished" podID="7c285da3-2e11-498d-94f1-76c0ce604553" containerID="78b5c1012635d39793a7b978cf50e9388d5edea5afd509bfce4fa68c48bf2e5a" exitCode=255 Nov 02 01:24:43.469790 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.469763 5044 generic.go:356] "Generic (PLEG): container finished" podID="9b91ad19-9236-4395-9e54-09470929bf35" containerID="bc228dde72cff8ba3dac2557a0acf99404a1244cf560011650a2df2584335c00" exitCode=0 Nov 02 01:24:43.469879 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.469791 5044 generic.go:356] "Generic (PLEG): container finished" podID="9b91ad19-9236-4395-9e54-09470929bf35" containerID="9921ab932f988037f47c0de6df7bbcb96fb3c29dbbd89ecc8ad4da7051f22bfc" exitCode=0 Nov 02 01:24:43.469879 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.469802 5044 generic.go:356] "Generic (PLEG): container finished" podID="9b91ad19-9236-4395-9e54-09470929bf35" containerID="52cfaac52b7d581bcf70568552e2938c6770de749284ffc6344f31fd3cbe0b29" exitCode=0 Nov 02 01:24:43.469968 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.469864 5044 generic.go:356] "Generic (PLEG): container finished" podID="9b91ad19-9236-4395-9e54-09470929bf35" containerID="ebf1caa1d56ea1dec1b3f2840f28ec0cc26651f2425fa52f99d4429b3d42625d" exitCode=0 Nov 02 01:24:43.469968 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.469892 5044 generic.go:356] "Generic (PLEG): container finished" podID="9b91ad19-9236-4395-9e54-09470929bf35" containerID="f19f28adba58efff0ae92b9429561293b64faa1dcf6481af6591de81d51ce982" exitCode=0 Nov 02 01:24:43.469968 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.469902 5044 generic.go:356] "Generic (PLEG): container finished" podID="9b91ad19-9236-4395-9e54-09470929bf35" containerID="83ef0f032e62a4c342d3b8a2fa490baa32f5b8ed7e8b550b6acb686623c8836a" exitCode=0 Nov 02 01:24:43.483891 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.483868 5044 generic.go:356] "Generic (PLEG): container finished" podID="1c52cca1e001b790fd8c346cdd80a9e7" containerID="604c7bf3cede74e00e8560f530e4c9bdb32b7262cd71a8a5a47cee2083c835f0" exitCode=0 Nov 02 01:24:43.513773 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.513754 5044 manager.go:324] Recovery completed Nov 02 01:24:43.516049 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:43.516027 5044 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-6.us-west-1.compute.internal\" not found" Nov 02 01:24:43.532436 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.532419 5044 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:24:43.535829 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.535809 5044 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:24:43.535927 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.535842 5044 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:24:43.535927 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.535852 5044 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 01:24:43.537952 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.537934 5044 cpu_manager.go:226] "Starting" policy="none" Nov 02 01:24:43.537952 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.537948 5044 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 01:24:43.537952 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.537950 5044 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:24:43.538073 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.537964 5044 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 01:24:43.538143 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.538126 5044 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 01:24:43.538184 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.538144 5044 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 01:24:43.538184 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.538161 5044 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 01:24:43.538184 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.538167 5044 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 01:24:43.538184 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.538172 5044 policy_none.go:50] "Start" Nov 02 01:24:43.538184 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.538180 5044 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 01:24:43.538320 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.538191 5044 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 01:24:43.538352 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.538340 5044 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 01:24:43.538352 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.538348 5044 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 01:24:43.538406 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.538353 5044 policy_none.go:44] "Start" Nov 02 01:24:43.538863 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.538847 5044 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:24:43.538916 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.538873 5044 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:24:43.538916 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.538883 5044 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 01:24:43.542784 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.542745 5044 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 01:24:43.544455 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:43.544438 5044 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 01:24:43.546359 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.546347 5044 manager.go:341] "Starting Device Plugin manager" Nov 02 01:24:43.546420 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:43.546372 5044 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 01:24:43.546420 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.546382 5044 server.go:87] "Starting device plugin registration server" Nov 02 01:24:43.546638 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.546627 5044 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 01:24:43.546702 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.546641 5044 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 01:24:43.546744 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.546727 5044 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 01:24:43.546853 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.546813 5044 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 01:24:43.546965 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.546857 5044 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 01:24:43.546965 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.546864 5044 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 01:24:43.547071 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.546916 5044 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T01:24:43.546846131Z","UUID":"1b8779cb-e711-45d1-9386-1b3385c861ae","Handler":null,"Name":"","Endpoint":""} Nov 02 01:24:43.548255 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.548237 5044 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 01:24:43.548345 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.548271 5044 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 01:24:43.548403 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:43.548347 5044 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 01:24:43.548403 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:43.548389 5044 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-6.us-west-1.compute.internal\" not found" Nov 02 01:24:43.604479 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.604452 5044 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:24:43.604584 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.604500 5044 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 01:24:43.604584 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.604499 5044 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:24:43.605295 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.605277 5044 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:24:43.647314 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.647289 5044 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 01:24:43.648472 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.648455 5044 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 01:24:43.648552 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.648486 5044 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 01:24:43.648552 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.648497 5044 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 01:24:43.648552 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.648541 5044 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:43.656415 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.656397 5044 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 01:24:43.745410 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.745342 5044 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal"] Nov 02 01:24:43.746076 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.746036 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal" event={"ID":"817dd2f8fadd21a6e3346c963f0534ec","Type":"ContainerStarted","Data":"729b6926b84ebd4b24e5750dfca74a28f4f321eb1f6a82601d39b4fa8be6a036"} Nov 02 01:24:43.746184 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.746081 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal" event={"ID":"817dd2f8fadd21a6e3346c963f0534ec","Type":"ContainerStarted","Data":"60ee38920eb119bf15b637d23b79c11689392dd6adbec04062c5121447e03e03"} Nov 02 01:24:43.746184 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.746107 5044 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:43.746318 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.746249 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" event={"ID":"1c52cca1e001b790fd8c346cdd80a9e7","Type":"ContainerStarted","Data":"17b51a45675b4dcb0bae224833d657cc961be6839ead272ca5fb690d2d35d12b"} Nov 02 01:24:43.746318 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.746284 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" event={"ID":"1c52cca1e001b790fd8c346cdd80a9e7","Type":"ContainerDied","Data":"604c7bf3cede74e00e8560f530e4c9bdb32b7262cd71a8a5a47cee2083c835f0"} Nov 02 01:24:43.746318 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.746305 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" event={"ID":"1c52cca1e001b790fd8c346cdd80a9e7","Type":"ContainerStarted","Data":"0a6d9848ff171086e151f397491a8f514dc8832592a239c33122c3284875456f"} Nov 02 01:24:43.746503 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.746359 5044 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:43.761954 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.761940 5044 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:43.762012 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.762002 5044 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:43.762588 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.762573 5044 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeNotReady" Nov 02 01:24:43.762689 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.762603 5044 setters.go:546] "Node became not ready" node="ip-10-0-129-6.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T01:24:43Z","lastTransitionTime":"2026-11-02T01:24:43Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 02 01:24:43.775155 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.775097 5044 nodeinfomanager.go:471] existing CSINode "ip-10-0-129-6.us-west-1.compute.internal" is owned by different node (oldNodeID="b33d76b9-d5a6-448d-a742-00502f87fdda", newNodeID=""), cleaning up... Nov 02 01:24:43.810680 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.810652 5044 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 01:24:43.810793 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:43.810712 5044 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:43.811222 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.811210 5044 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 01:24:43.811255 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:43.811238 5044 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:43.907030 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.906862 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c52cca1e001b790fd8c346cdd80a9e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal\" (UID: \"1c52cca1e001b790fd8c346cdd80a9e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:43.907030 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.906909 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/817dd2f8fadd21a6e3346c963f0534ec-config\") pod \"kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal\" (UID: \"817dd2f8fadd21a6e3346c963f0534ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:43.907030 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:43.906933 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c52cca1e001b790fd8c346cdd80a9e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal\" (UID: \"1c52cca1e001b790fd8c346cdd80a9e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:44.007528 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.007489 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/817dd2f8fadd21a6e3346c963f0534ec-config\") pod \"kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal\" (UID: \"817dd2f8fadd21a6e3346c963f0534ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:44.007528 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.007533 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c52cca1e001b790fd8c346cdd80a9e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal\" (UID: \"1c52cca1e001b790fd8c346cdd80a9e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:44.007757 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.007603 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/817dd2f8fadd21a6e3346c963f0534ec-config\") pod \"kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal\" (UID: \"817dd2f8fadd21a6e3346c963f0534ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:44.007757 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.007629 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c52cca1e001b790fd8c346cdd80a9e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal\" (UID: \"1c52cca1e001b790fd8c346cdd80a9e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:44.007757 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.007695 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c52cca1e001b790fd8c346cdd80a9e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal\" (UID: \"1c52cca1e001b790fd8c346cdd80a9e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:44.007757 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.007700 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c52cca1e001b790fd8c346cdd80a9e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal\" (UID: \"1c52cca1e001b790fd8c346cdd80a9e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:44.390565 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.390535 5044 apiserver.go:52] "Watching apiserver" Nov 02 01:24:44.504494 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.504469 5044 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:44.504641 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.504576 5044 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:44.547265 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.547238 5044 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 01:24:44.548009 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.547983 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d9wwq","openshift-ingress/router-default-6cb67dcd4c-zk2vs","openshift-kube-storage-version-migrator/migrator-66df5d496b-q29rd","openshift-kube-storage-version-migrator/migrator-66df5d496b-n5ppw","openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz","openshift-dns/dns-default-qmrv6","openshift-insights/periodic-gathering-5bwtl-qgqdr","kube-system/global-pull-secret-syncer-7szx6","openshift-insights/insights-runtime-extractor-r2hxq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal","kube-system/kas-connection-checker-685895c7c9-v8sdv","openshift-console-operator/console-operator-7cd56b8dbc-frfx9","openshift-image-registry/image-registry-69469b6ffb-qmvll","openshift-dns/node-resolver-ht26h","openshift-image-registry/node-ca-rrf9f","openshift-cluster-node-tuning-operator/tuned-gwkc5","openshift-multus/network-metrics-daemon-flwvx","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq","openshift-ovn-kubernetes/ovnkube-node-qzhvs","openshift-service-ca/service-ca-5fffc97bdf-qkqj6","openshift-network-operator/iptables-alerter-zzm69","openshift-insights/insights-operator-7c4986d449-tgsq4","openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk","openshift-ingress-canary/ingress-canary-9sgpk","openshift-image-registry/image-registry-5954c6588f-7h92t","kube-system/kas-connection-checker-685895c7c9-xmlts","kube-system/konnectivity-agent-7mk45","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl","openshift-network-console/networking-console-plugin-86c478bb88-ntpqz","kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal","openshift-multus/multus-wzzsb","openshift-network-diagnostics/network-check-target-kgbsx","openshift-multus/multus-additional-cni-plugins-mgjgw","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9","openshift-network-diagnostics/network-check-source-6f78d59f99-58fr4","kube-system/kas-connection-checker-685895c7c9-zlq59"] Nov 02 01:24:44.549535 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.549514 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:44.550380 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.550353 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:44.551909 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.551889 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:44.552376 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.552356 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:44.553138 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.553119 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:44.554600 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.554583 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:44.562534 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.562516 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:44.563517 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.563501 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:44.564076 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.564060 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:44.566569 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.566551 5044 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 01:24:44.566651 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:44.566600 5044 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:44.567029 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.567017 5044 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 01:24:44.567099 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:44.567046 5044 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-6.us-west-1.compute.internal" Nov 02 01:24:44.605736 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.605713 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 01:24:44.605736 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.605716 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 01:24:44.605874 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.605737 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 01:24:44.605874 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.605794 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 01:24:44.605874 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.605820 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 01:24:44.605874 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.605851 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:24:44.606083 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.605891 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 01:24:44.606083 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.605945 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 01:24:44.606083 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.605952 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 01:24:44.606204 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.605956 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 01:24:44.606204 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.606000 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 01:24:44.606638 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.606620 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 01:24:44.608168 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.608147 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 01:24:44.609464 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.609447 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 01:24:44.609532 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.609490 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 01:24:44.609650 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.609631 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 01:24:44.609742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.609721 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 01:24:44.609742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.609733 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 01:24:44.609884 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.609732 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 01:24:44.609884 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.609751 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 01:24:44.609884 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.609765 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 01:24:44.609884 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.609797 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 01:24:44.609884 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.609798 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:24:44.609884 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.609833 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:24:44.609884 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.609844 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 01:24:44.609884 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.609856 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:24:44.609884 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.609861 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 01:24:44.610270 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.609894 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 01:24:44.610270 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.609896 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 01:24:44.610270 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.609927 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:24:44.610270 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.609960 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 01:24:44.610270 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.610177 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 01:24:44.610582 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.610557 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jvpgw\"" Nov 02 01:24:44.610582 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.610562 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 01:24:44.610856 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.610566 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 01:24:44.610856 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.610601 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 01:24:44.610856 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.610608 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 01:24:44.610856 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.610607 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 01:24:44.610856 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.610628 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 01:24:44.610856 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.610617 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 01:24:44.610856 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.610697 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 01:24:44.610856 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.610700 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 01:24:44.610856 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.610665 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 01:24:44.610856 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.610786 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 01:24:44.610856 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.610836 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 01:24:44.618336 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.618321 5044 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 01:24:44.656534 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.656473 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 01:24:44.656534 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.656477 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p5zpq\"" Nov 02 01:24:44.656534 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.656501 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-vkh6q\"" Nov 02 01:24:44.656534 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.656511 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2dvkb\"" Nov 02 01:24:44.656969 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.656740 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-82k56\"" Nov 02 01:24:44.656969 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.656894 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 01:24:44.658580 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.658503 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 01:24:44.658762 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.658747 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 01:24:44.658852 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.658836 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 01:24:44.659033 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.659016 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 01:24:44.659095 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.658791 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 01:24:44.659095 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.659082 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 01:24:44.659184 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.659017 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 01:24:44.659184 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.658547 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 01:24:44.659270 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.658558 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pw4q8\"" Nov 02 01:24:44.659322 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.658784 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:24:44.659322 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.659018 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-gsg88\"" Nov 02 01:24:44.659456 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.659441 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 01:24:44.659720 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.659704 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 01:24:44.659997 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.659970 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 01:24:44.660074 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.660000 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 01:24:44.661089 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.661062 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 01:24:44.662159 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.662132 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 01:24:44.704554 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.704534 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 01:24:44.704721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.704558 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 01:24:44.704721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.704650 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 01:24:44.705126 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705104 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 01:24:44.705229 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705147 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 01:24:44.705229 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705186 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 01:24:44.705330 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705308 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:24:44.705377 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705348 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wwr4r\"" Nov 02 01:24:44.705377 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705360 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gh69v\"" Nov 02 01:24:44.705439 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705407 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6qqtd\"" Nov 02 01:24:44.705502 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705489 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 01:24:44.705551 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705505 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q5j6l\"" Nov 02 01:24:44.705551 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705513 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 01:24:44.705636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705567 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pt8bk\"" Nov 02 01:24:44.705636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705618 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 01:24:44.705636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705617 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cpkxg\"" Nov 02 01:24:44.705636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705628 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 01:24:44.705810 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705646 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 01:24:44.705810 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705659 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 01:24:44.705810 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705660 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kv2jl\"" Nov 02 01:24:44.705810 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705696 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-klgj9\"" Nov 02 01:24:44.705810 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705705 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 01:24:44.705810 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705700 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4gwsn\"" Nov 02 01:24:44.705810 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705706 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 01:24:44.705810 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705720 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 01:24:44.705810 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705720 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 01:24:44.705810 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705733 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 01:24:44.705810 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705743 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 01:24:44.706111 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705831 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tl75j\"" Nov 02 01:24:44.706111 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705870 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l67lg\"" Nov 02 01:24:44.706111 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705918 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 01:24:44.706111 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.705967 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 01:24:44.706111 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.706015 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 01:24:44.706111 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.706041 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 01:24:44.706111 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.706074 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 01:24:44.706296 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.706118 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 01:24:44.706296 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.706170 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qtccq\"" Nov 02 01:24:44.706296 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.706212 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wfbzc\"" Nov 02 01:24:44.710995 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.710974 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91cb4fd5-ed27-4a79-b172-be80b203bf0f-serving-cert\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:44.711080 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711006 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a128d9e6-7be3-4c73-88f3-ba6b9ea59971-serviceca\") pod \"node-ca-rrf9f\" (UID: \"a128d9e6-7be3-4c73-88f3-ba6b9ea59971\") " pod="openshift-image-registry/node-ca-rrf9f" Nov 02 01:24:44.711080 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711029 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z8mm\" (UniqueName: \"kubernetes.io/projected/2e031e5c-cb8a-42b9-b334-1cbe61a9c59f-kube-api-access-6z8mm\") pod \"kas-connection-checker-685895c7c9-zlq59\" (UID: \"2e031e5c-cb8a-42b9-b334-1cbe61a9c59f\") " pod="kube-system/kas-connection-checker-685895c7c9-zlq59" Nov 02 01:24:44.711080 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711059 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ntpqz\" (UID: \"fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:44.711230 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711211 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/91cb4fd5-ed27-4a79-b172-be80b203bf0f-serving-cert\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:44.711287 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711230 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-socket-dir-parent\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.711287 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711269 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drbjx\" (UniqueName: \"kubernetes.io/projected/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-kube-api-access-drbjx\") pod \"periodic-gathering-5bwtl-qgqdr\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:44.711368 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711345 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc4n8\" (UniqueName: \"kubernetes.io/projected/476d365d-dd0a-41e6-a186-5e92551be763-kube-api-access-dc4n8\") pod \"migrator-66df5d496b-n5ppw\" (UID: \"476d365d-dd0a-41e6-a186-5e92551be763\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n5ppw" Nov 02 01:24:44.711421 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711399 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a128d9e6-7be3-4c73-88f3-ba6b9ea59971-serviceca\") pod \"node-ca-rrf9f\" (UID: \"a128d9e6-7be3-4c73-88f3-ba6b9ea59971\") " pod="openshift-image-registry/node-ca-rrf9f" Nov 02 01:24:44.711454 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711416 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-etc-selinux\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:44.711454 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711446 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54db4b99-9007-429a-a267-b4924a8729e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:44.711538 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711503 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-kubelet\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.711576 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711535 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-run\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.711576 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711570 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b91ad19-9236-4395-9e54-09470929bf35-cnibin\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:44.711636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711595 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b91ad19-9236-4395-9e54-09470929bf35-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:44.711636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711628 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-certificates\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:44.711700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711685 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6sdmz\" (UID: \"59ccadf3-202d-481d-8c8b-c4fa86c92b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:44.711772 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711756 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-sys\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.711833 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711804 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-certificates\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:44.711868 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711806 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-var-lib-openvswitch\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.711868 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711858 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a128d9e6-7be3-4c73-88f3-ba6b9ea59971-host\") pod \"node-ca-rrf9f\" (UID: \"a128d9e6-7be3-4c73-88f3-ba6b9ea59971\") " pod="openshift-image-registry/node-ca-rrf9f" Nov 02 01:24:44.711952 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711880 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79f97db3-814b-4849-ab8d-305dbcd3449c-dbus\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:44.711952 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711898 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4abd5311-4a92-4ac5-8aa0-779dd8774577-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtll9\" (UID: \"4abd5311-4a92-4ac5-8aa0-779dd8774577\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" Nov 02 01:24:44.711952 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711927 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-socket-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:44.711952 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.711931 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6sdmz\" (UID: \"59ccadf3-202d-481d-8c8b-c4fa86c92b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:44.712084 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712010 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-daemon-config\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.712084 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712048 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4abd5311-4a92-4ac5-8aa0-779dd8774577-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtll9\" (UID: \"4abd5311-4a92-4ac5-8aa0-779dd8774577\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" Nov 02 01:24:44.712084 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712052 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91cb4fd5-ed27-4a79-b172-be80b203bf0f-tmp\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:44.712254 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712114 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-run-netns\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.712254 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712146 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-metrics-tls\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:44.712254 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712163 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:44.712254 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712168 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91cb4fd5-ed27-4a79-b172-be80b203bf0f-tmp\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:44.712254 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712211 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4abd5311-4a92-4ac5-8aa0-779dd8774577-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtll9\" (UID: \"4abd5311-4a92-4ac5-8aa0-779dd8774577\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" Nov 02 01:24:44.712254 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712230 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-cni-dir\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.712424 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712257 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.712424 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712308 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35ae4588-f166-4563-aae7-b34b5b053d0f-serving-cert\") pod \"service-ca-operator-544d749c8c-868jk\" (UID: \"35ae4588-f166-4563-aae7-b34b5b053d0f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" Nov 02 01:24:44.712424 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712333 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-daemon-config\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.712424 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712342 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j98n8\" (UniqueName: \"kubernetes.io/projected/a128d9e6-7be3-4c73-88f3-ba6b9ea59971-kube-api-access-j98n8\") pod \"node-ca-rrf9f\" (UID: \"a128d9e6-7be3-4c73-88f3-ba6b9ea59971\") " pod="openshift-image-registry/node-ca-rrf9f" Nov 02 01:24:44.712424 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712358 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-cni-binary-copy\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.712424 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712390 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-var-lib-cni-bin\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.712593 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712418 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b083c3f1-9c74-4313-a939-ff8aa5769c8d-ovn-node-metrics-cert\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.712593 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712478 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4abd5311-4a92-4ac5-8aa0-779dd8774577-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtll9\" (UID: \"4abd5311-4a92-4ac5-8aa0-779dd8774577\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" Nov 02 01:24:44.712593 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712479 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-trusted-ca\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:44.712593 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712421 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79f97db3-814b-4849-ab8d-305dbcd3449c-original-pull-secret\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:44.712593 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712518 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-hostroot\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.712593 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712513 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-cni-binary-copy\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.712593 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712542 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-slash\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.712593 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712569 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b083c3f1-9c74-4313-a939-ff8aa5769c8d-env-overrides\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.712933 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712637 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp982\" (UniqueName: \"kubernetes.io/projected/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-kube-api-access-xp982\") pod \"cluster-monitoring-operator-884576c57-6sdmz\" (UID: \"59ccadf3-202d-481d-8c8b-c4fa86c92b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:44.712933 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712638 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35ae4588-f166-4563-aae7-b34b5b053d0f-serving-cert\") pod \"service-ca-operator-544d749c8c-868jk\" (UID: \"35ae4588-f166-4563-aae7-b34b5b053d0f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" Nov 02 01:24:44.712933 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712663 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-ca-trust-extracted\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:44.712933 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712815 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b083c3f1-9c74-4313-a939-ff8aa5769c8d-env-overrides\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.712933 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712827 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-ca-trust-extracted\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:44.712933 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712801 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bb2a916a-fc15-4bdc-b28a-c2473a543ce5-agent-certs\") pod \"konnectivity-agent-7mk45\" (UID: \"bb2a916a-fc15-4bdc-b28a-c2473a543ce5\") " pod="kube-system/konnectivity-agent-7mk45" Nov 02 01:24:44.712933 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712872 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-trusted-ca\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:44.712933 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712874 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-run-netns\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.712933 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712815 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b083c3f1-9c74-4313-a939-ff8aa5769c8d-ovn-node-metrics-cert\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.713217 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712927 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-etc-openvswitch\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.713217 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.712973 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-tmp-dir\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:44.713217 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713012 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-sys-fs\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:44.713217 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713023 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-tmp-dir\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:44.713217 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713049 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-cnibin\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.713217 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713078 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg9vq\" (UniqueName: \"kubernetes.io/projected/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-kube-api-access-jg9vq\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.713217 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713095 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bb2a916a-fc15-4bdc-b28a-c2473a543ce5-agent-certs\") pod \"konnectivity-agent-7mk45\" (UID: \"bb2a916a-fc15-4bdc-b28a-c2473a543ce5\") " pod="kube-system/konnectivity-agent-7mk45" Nov 02 01:24:44.713217 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713121 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-cni-bin\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.713217 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713201 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b91ad19-9236-4395-9e54-09470929bf35-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:44.713470 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713228 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1ed8a22-43d0-4ca3-9f7b-900231ad3685-hosts-file\") pod \"node-resolver-ht26h\" (UID: \"a1ed8a22-43d0-4ca3-9f7b-900231ad3685\") " pod="openshift-dns/node-resolver-ht26h" Nov 02 01:24:44.713470 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713323 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:44.713470 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713362 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b91ad19-9236-4395-9e54-09470929bf35-system-cni-dir\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:44.713470 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713384 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/35ae4588-f166-4563-aae7-b34b5b053d0f-config\") pod \"service-ca-operator-544d749c8c-868jk\" (UID: \"35ae4588-f166-4563-aae7-b34b5b053d0f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" Nov 02 01:24:44.713470 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713392 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b91ad19-9236-4395-9e54-09470929bf35-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:44.713470 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713414 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-trusted-ca-bundle\") pod \"periodic-gathering-5bwtl-qgqdr\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:44.713470 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713445 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-kubernetes\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.713683 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713503 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/673cb927-a5fc-4237-9b4c-fb196801d70c-iptables-alerter-script\") pod \"iptables-alerter-zzm69\" (UID: \"673cb927-a5fc-4237-9b4c-fb196801d70c\") " pod="openshift-network-operator/iptables-alerter-zzm69" Nov 02 01:24:44.713683 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713529 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/673cb927-a5fc-4237-9b4c-fb196801d70c-host-slash\") pod \"iptables-alerter-zzm69\" (UID: \"673cb927-a5fc-4237-9b4c-fb196801d70c\") " pod="openshift-network-operator/iptables-alerter-zzm69" Nov 02 01:24:44.713683 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713570 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/35ae4588-f166-4563-aae7-b34b5b053d0f-config\") pod \"service-ca-operator-544d749c8c-868jk\" (UID: \"35ae4588-f166-4563-aae7-b34b5b053d0f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" Nov 02 01:24:44.713683 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713587 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-log-socket\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.713683 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713623 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgsc7\" (UniqueName: \"kubernetes.io/projected/e6c17513-0478-4a05-94dd-0cdad5facf06-kube-api-access-rgsc7\") pod \"migrator-66df5d496b-q29rd\" (UID: \"e6c17513-0478-4a05-94dd-0cdad5facf06\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q29rd" Nov 02 01:24:44.713683 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713650 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bb2a916a-fc15-4bdc-b28a-c2473a543ce5-konnectivity-ca\") pod \"konnectivity-agent-7mk45\" (UID: \"bb2a916a-fc15-4bdc-b28a-c2473a543ce5\") " pod="kube-system/konnectivity-agent-7mk45" Nov 02 01:24:44.713905 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713701 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:44.713905 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713730 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdnrv\" (UniqueName: \"kubernetes.io/projected/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-kube-api-access-kdnrv\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:44.713905 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713755 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e79ec42c-3696-40e7-94de-57aad9a9f92c-cert\") pod \"ingress-canary-9sgpk\" (UID: \"e79ec42c-3696-40e7-94de-57aad9a9f92c\") " pod="openshift-ingress-canary/ingress-canary-9sgpk" Nov 02 01:24:44.713905 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713761 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/673cb927-a5fc-4237-9b4c-fb196801d70c-iptables-alerter-script\") pod \"iptables-alerter-zzm69\" (UID: \"673cb927-a5fc-4237-9b4c-fb196801d70c\") " pod="openshift-network-operator/iptables-alerter-zzm69" Nov 02 01:24:44.713905 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713756 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-trusted-ca-bundle\") pod \"periodic-gathering-5bwtl-qgqdr\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:44.713905 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713890 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bb2a916a-fc15-4bdc-b28a-c2473a543ce5-konnectivity-ca\") pod \"konnectivity-agent-7mk45\" (UID: \"bb2a916a-fc15-4bdc-b28a-c2473a543ce5\") " pod="kube-system/konnectivity-agent-7mk45" Nov 02 01:24:44.714165 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713909 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-archives-path\") pod \"periodic-gathering-5bwtl-qgqdr\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:44.714165 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713960 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4wlx\" (UniqueName: \"kubernetes.io/projected/a1ed8a22-43d0-4ca3-9f7b-900231ad3685-kube-api-access-b4wlx\") pod \"node-resolver-ht26h\" (UID: \"a1ed8a22-43d0-4ca3-9f7b-900231ad3685\") " pod="openshift-dns/node-resolver-ht26h" Nov 02 01:24:44.714165 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.713988 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-archives-path\") pod \"periodic-gathering-5bwtl-qgqdr\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:44.714165 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714005 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq7hj\" (UniqueName: \"kubernetes.io/projected/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-kube-api-access-rq7hj\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:44.714165 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714022 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e79ec42c-3696-40e7-94de-57aad9a9f92c-cert\") pod \"ingress-canary-9sgpk\" (UID: \"e79ec42c-3696-40e7-94de-57aad9a9f92c\") " pod="openshift-ingress-canary/ingress-canary-9sgpk" Nov 02 01:24:44.714165 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714113 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79f97db3-814b-4849-ab8d-305dbcd3449c-kubelet-config\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:44.714397 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714188 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:44.714397 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714223 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-default-certificate\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:44.714397 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714285 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/54db4b99-9007-429a-a267-b4924a8729e6-data-volume\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:44.714397 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714347 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/54db4b99-9007-429a-a267-b4924a8729e6-data-volume\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:44.714397 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714358 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-systemd-units\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.714397 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714390 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-cni-netd\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.714666 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714410 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/028b56de-c4b9-47c1-8bfa-761877176429-signing-cabundle\") pod \"service-ca-5fffc97bdf-qkqj6\" (UID: \"028b56de-c4b9-47c1-8bfa-761877176429\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qkqj6" Nov 02 01:24:44.714666 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714433 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4q2q\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-kube-api-access-r4q2q\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:44.714666 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714454 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-default-certificate\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:44.714666 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714477 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6sdmz\" (UID: \"59ccadf3-202d-481d-8c8b-c4fa86c92b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:44.714666 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714513 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-systemd\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.714666 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714536 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-host\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.714666 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714562 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-var-lib-cni-multus\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.714666 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714624 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b083c3f1-9c74-4313-a939-ff8aa5769c8d-ovnkube-script-lib\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.714666 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714647 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/028b56de-c4b9-47c1-8bfa-761877176429-signing-cabundle\") pod \"service-ca-5fffc97bdf-qkqj6\" (UID: \"028b56de-c4b9-47c1-8bfa-761877176429\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qkqj6" Nov 02 01:24:44.714666 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714650 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-stats-auth\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:44.715145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714690 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:44.715145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714718 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91cb4fd5-ed27-4a79-b172-be80b203bf0f-service-ca-bundle\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:44.715145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714744 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqppx\" (UniqueName: \"kubernetes.io/projected/9b91ad19-9236-4395-9e54-09470929bf35-kube-api-access-lqppx\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:44.715145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714787 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-lib-modules\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.715145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714810 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b083c3f1-9c74-4313-a939-ff8aa5769c8d-ovnkube-script-lib\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.715145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714821 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4h6s\" (UniqueName: \"kubernetes.io/projected/028b56de-c4b9-47c1-8bfa-761877176429-kube-api-access-k4h6s\") pod \"service-ca-5fffc97bdf-qkqj6\" (UID: \"028b56de-c4b9-47c1-8bfa-761877176429\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qkqj6" Nov 02 01:24:44.715145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714858 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-stats-auth\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:44.715145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714864 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c285da3-2e11-498d-94f1-76c0ce604553-trusted-ca\") pod \"console-operator-7cd56b8dbc-frfx9\" (UID: \"7c285da3-2e11-498d-94f1-76c0ce604553\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:44.715145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714921 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91cb4fd5-ed27-4a79-b172-be80b203bf0f-service-ca-bundle\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:44.715145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.714969 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:44.715145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715036 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e534da40-0733-4670-ba7a-839e0e4b1b4f-tmp\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.715145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715071 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-var-lib-kubelet\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.715145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715109 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e534da40-0733-4670-ba7a-839e0e4b1b4f-tmp\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.715145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715117 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-etc-kubernetes\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.715145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715140 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-run-openvswitch\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715165 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b083c3f1-9c74-4313-a939-ff8aa5769c8d-ovnkube-config\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715196 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c2b846b-5c8b-4982-abc9-809fdbffe82d-ca-trust-extracted\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715227 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c285da3-2e11-498d-94f1-76c0ce604553-trusted-ca\") pod \"console-operator-7cd56b8dbc-frfx9\" (UID: \"7c285da3-2e11-498d-94f1-76c0ce604553\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715229 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-service-ca-bundle\") pod \"periodic-gathering-5bwtl-qgqdr\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715273 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c2b846b-5c8b-4982-abc9-809fdbffe82d-ca-trust-extracted\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715286 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ed8a22-43d0-4ca3-9f7b-900231ad3685-tmp-dir\") pod \"node-resolver-ht26h\" (UID: \"a1ed8a22-43d0-4ca3-9f7b-900231ad3685\") " pod="openshift-dns/node-resolver-ht26h" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715315 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8wfd\" (UniqueName: \"kubernetes.io/projected/7c285da3-2e11-498d-94f1-76c0ce604553-kube-api-access-v8wfd\") pod \"console-operator-7cd56b8dbc-frfx9\" (UID: \"7c285da3-2e11-498d-94f1-76c0ce604553\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715336 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1ed8a22-43d0-4ca3-9f7b-900231ad3685-tmp-dir\") pod \"node-resolver-ht26h\" (UID: \"a1ed8a22-43d0-4ca3-9f7b-900231ad3685\") " pod="openshift-dns/node-resolver-ht26h" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715341 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh2r4\" (UniqueName: \"kubernetes.io/projected/d628518e-2ab7-4acb-acf7-6fc96f359a64-kube-api-access-rh2r4\") pod \"network-metrics-daemon-flwvx\" (UID: \"d628518e-2ab7-4acb-acf7-6fc96f359a64\") " pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715366 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-service-ca-bundle\") pod \"periodic-gathering-5bwtl-qgqdr\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715373 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-nginx-conf\") pod \"networking-console-plugin-86c478bb88-ntpqz\" (UID: \"fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715407 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54db4b99-9007-429a-a267-b4924a8729e6-crio-socket\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715429 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b083c3f1-9c74-4313-a939-ff8aa5769c8d-ovnkube-config\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715433 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b91ad19-9236-4395-9e54-09470929bf35-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715459 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c2b846b-5c8b-4982-abc9-809fdbffe82d-installation-pull-secrets\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715486 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c285da3-2e11-498d-94f1-76c0ce604553-serving-cert\") pod \"console-operator-7cd56b8dbc-frfx9\" (UID: \"7c285da3-2e11-498d-94f1-76c0ce604553\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715513 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-bound-sa-token\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715542 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-nginx-conf\") pod \"networking-console-plugin-86c478bb88-ntpqz\" (UID: \"fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715550 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-modprobe-d\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715574 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/91cb4fd5-ed27-4a79-b172-be80b203bf0f-snapshots\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715634 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b91ad19-9236-4395-9e54-09470929bf35-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715651 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c285da3-2e11-498d-94f1-76c0ce604553-serving-cert\") pod \"console-operator-7cd56b8dbc-frfx9\" (UID: \"7c285da3-2e11-498d-94f1-76c0ce604553\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715655 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c2b846b-5c8b-4982-abc9-809fdbffe82d-installation-pull-secrets\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715661 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/91cb4fd5-ed27-4a79-b172-be80b203bf0f-snapshots\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715700 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxmrq\" (UniqueName: \"kubernetes.io/projected/363dffa0-70cf-4fe0-996f-3c4d2f3b1a50-kube-api-access-bxmrq\") pod \"volume-data-source-validator-58c9f5f46d-d9wwq\" (UID: \"363dffa0-70cf-4fe0-996f-3c4d2f3b1a50\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d9wwq" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715734 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-run-k8s-cni-cncf-io\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715759 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crplh\" (UniqueName: \"kubernetes.io/projected/b083c3f1-9c74-4313-a939-ff8aa5769c8d-kube-api-access-crplh\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.715809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715786 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c2b846b-5c8b-4982-abc9-809fdbffe82d-trusted-ca\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715877 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjz2b\" (UniqueName: \"kubernetes.io/projected/8618e1e8-4d71-4f7a-b2a1-1b950190f110-kube-api-access-wjz2b\") pod \"kas-connection-checker-685895c7c9-xmlts\" (UID: \"8618e1e8-4d71-4f7a-b2a1-1b950190f110\") " pod="kube-system/kas-connection-checker-685895c7c9-xmlts" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715909 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-image-registry-private-configuration\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715937 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-sysconfig\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715965 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tghz\" (UniqueName: \"kubernetes.io/projected/e79ec42c-3696-40e7-94de-57aad9a9f92c-kube-api-access-4tghz\") pod \"ingress-canary-9sgpk\" (UID: \"e79ec42c-3696-40e7-94de-57aad9a9f92c\") " pod="openshift-ingress-canary/ingress-canary-9sgpk" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.715992 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zjgc\" (UniqueName: \"kubernetes.io/projected/e534da40-0733-4670-ba7a-839e0e4b1b4f-kube-api-access-5zjgc\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716037 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-device-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716065 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-conf-dir\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716095 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-862mp\" (UniqueName: \"kubernetes.io/projected/91cb4fd5-ed27-4a79-b172-be80b203bf0f-kube-api-access-862mp\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716121 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716151 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c2b846b-5c8b-4982-abc9-809fdbffe82d-trusted-ca\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716186 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/028b56de-c4b9-47c1-8bfa-761877176429-signing-key\") pod \"service-ca-5fffc97bdf-qkqj6\" (UID: \"028b56de-c4b9-47c1-8bfa-761877176429\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qkqj6" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716204 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b91ad19-9236-4395-9e54-09470929bf35-cni-binary-copy\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716311 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-image-registry-private-configuration\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716322 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b91ad19-9236-4395-9e54-09470929bf35-cni-binary-copy\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716447 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c2b846b-5c8b-4982-abc9-809fdbffe82d-image-registry-private-configuration\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716452 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/028b56de-c4b9-47c1-8bfa-761877176429-signing-key\") pod \"service-ca-5fffc97bdf-qkqj6\" (UID: \"028b56de-c4b9-47c1-8bfa-761877176429\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qkqj6" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716495 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-bound-sa-token\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716537 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-sysctl-conf\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716634 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57blk\" (UniqueName: \"kubernetes.io/projected/54db4b99-9007-429a-a267-b4924a8729e6-kube-api-access-57blk\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716656 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44rfr\" (UniqueName: \"kubernetes.io/projected/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-kube-api-access-44rfr\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716684 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c2b846b-5c8b-4982-abc9-809fdbffe82d-image-registry-private-configuration\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716723 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-installation-pull-secrets\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716759 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnz7p\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-kube-api-access-tnz7p\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716789 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh798\" (UniqueName: \"kubernetes.io/projected/eb8ad2e6-7e4f-4c0a-9ea3-99ed98344ba6-kube-api-access-xh798\") pod \"kas-connection-checker-685895c7c9-v8sdv\" (UID: \"eb8ad2e6-7e4f-4c0a-9ea3-99ed98344ba6\") " pod="kube-system/kas-connection-checker-685895c7c9-v8sdv" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716817 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-system-cni-dir\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716844 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-run-systemd\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716887 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s6ct\" (UniqueName: \"kubernetes.io/projected/35ae4588-f166-4563-aae7-b34b5b053d0f-kube-api-access-6s6ct\") pod \"service-ca-operator-544d749c8c-868jk\" (UID: \"35ae4588-f166-4563-aae7-b34b5b053d0f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716902 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-installation-pull-secrets\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716915 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c285da3-2e11-498d-94f1-76c0ce604553-config\") pod \"console-operator-7cd56b8dbc-frfx9\" (UID: \"7c285da3-2e11-498d-94f1-76c0ce604553\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:44.716920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716941 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-config-volume\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716969 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-certificates\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.716994 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-sysctl-d\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717058 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91cb4fd5-ed27-4a79-b172-be80b203bf0f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717093 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c285da3-2e11-498d-94f1-76c0ce604553-config\") pod \"console-operator-7cd56b8dbc-frfx9\" (UID: \"7c285da3-2e11-498d-94f1-76c0ce604553\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717154 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-certificates\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717168 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-tuned\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717159 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-config-volume\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717246 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs\") pod \"network-metrics-daemon-flwvx\" (UID: \"d628518e-2ab7-4acb-acf7-6fc96f359a64\") " pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717269 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-tuned\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717268 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-724nj\" (UniqueName: \"kubernetes.io/projected/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-kube-api-access-724nj\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717305 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717336 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dzj7\" (UniqueName: \"kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7\") pod \"network-check-target-kgbsx\" (UID: \"85784876-9397-4bec-8ade-878a15c09f40\") " pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717405 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-var-lib-kubelet\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717427 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91cb4fd5-ed27-4a79-b172-be80b203bf0f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717441 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrnrh\" (UniqueName: \"kubernetes.io/projected/673cb927-a5fc-4237-9b4c-fb196801d70c-kube-api-access-xrnrh\") pod \"iptables-alerter-zzm69\" (UID: \"673cb927-a5fc-4237-9b4c-fb196801d70c\") " pod="openshift-network-operator/iptables-alerter-zzm69" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717493 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-node-log\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717519 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-run-ovn-kubernetes\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717525 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717552 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvshf\" (UniqueName: \"kubernetes.io/projected/2c8cbfce-aaf5-4362-930e-b15539b2dbc4-kube-api-access-jvshf\") pod \"network-check-source-6f78d59f99-58fr4\" (UID: \"2c8cbfce-aaf5-4362-930e-b15539b2dbc4\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-58fr4" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717585 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpwzf\" (UniqueName: \"kubernetes.io/projected/4abd5311-4a92-4ac5-8aa0-779dd8774577-kube-api-access-tpwzf\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtll9\" (UID: \"4abd5311-4a92-4ac5-8aa0-779dd8774577\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717709 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-registration-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717736 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717762 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-run-multus-certs\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.718004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.717799 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-run-ovn\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.756277 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.756242 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:44.771282 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.771243 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc4n8\" (UniqueName: \"kubernetes.io/projected/476d365d-dd0a-41e6-a186-5e92551be763-kube-api-access-dc4n8\") pod \"migrator-66df5d496b-n5ppw\" (UID: \"476d365d-dd0a-41e6-a186-5e92551be763\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n5ppw" Nov 02 01:24:44.771387 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.771310 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z8mm\" (UniqueName: \"kubernetes.io/projected/2e031e5c-cb8a-42b9-b334-1cbe61a9c59f-kube-api-access-6z8mm\") pod \"kas-connection-checker-685895c7c9-zlq59\" (UID: \"2e031e5c-cb8a-42b9-b334-1cbe61a9c59f\") " pod="kube-system/kas-connection-checker-685895c7c9-zlq59" Nov 02 01:24:44.771387 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.771341 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drbjx\" (UniqueName: \"kubernetes.io/projected/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-kube-api-access-drbjx\") pod \"periodic-gathering-5bwtl-qgqdr\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:24:44.777951 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.777925 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgsc7\" (UniqueName: \"kubernetes.io/projected/e6c17513-0478-4a05-94dd-0cdad5facf06-kube-api-access-rgsc7\") pod \"migrator-66df5d496b-q29rd\" (UID: \"e6c17513-0478-4a05-94dd-0cdad5facf06\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q29rd" Nov 02 01:24:44.778050 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.777977 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j98n8\" (UniqueName: \"kubernetes.io/projected/a128d9e6-7be3-4c73-88f3-ba6b9ea59971-kube-api-access-j98n8\") pod \"node-ca-rrf9f\" (UID: \"a128d9e6-7be3-4c73-88f3-ba6b9ea59971\") " pod="openshift-image-registry/node-ca-rrf9f" Nov 02 01:24:44.778110 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.778094 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg9vq\" (UniqueName: \"kubernetes.io/projected/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-kube-api-access-jg9vq\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.778156 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.778140 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdnrv\" (UniqueName: \"kubernetes.io/projected/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-kube-api-access-kdnrv\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:44.778205 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.778184 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp982\" (UniqueName: \"kubernetes.io/projected/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-kube-api-access-xp982\") pod \"cluster-monitoring-operator-884576c57-6sdmz\" (UID: \"59ccadf3-202d-481d-8c8b-c4fa86c92b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:44.780869 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.780849 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57blk\" (UniqueName: \"kubernetes.io/projected/54db4b99-9007-429a-a267-b4924a8729e6-kube-api-access-57blk\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:44.781106 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.781089 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxmrq\" (UniqueName: \"kubernetes.io/projected/363dffa0-70cf-4fe0-996f-3c4d2f3b1a50-kube-api-access-bxmrq\") pod \"volume-data-source-validator-58c9f5f46d-d9wwq\" (UID: \"363dffa0-70cf-4fe0-996f-3c4d2f3b1a50\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-d9wwq" Nov 02 01:24:44.782843 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.782826 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crplh\" (UniqueName: \"kubernetes.io/projected/b083c3f1-9c74-4313-a939-ff8aa5769c8d-kube-api-access-crplh\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.783317 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.783300 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44rfr\" (UniqueName: \"kubernetes.io/projected/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-kube-api-access-44rfr\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:44.783482 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.783462 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-bound-sa-token\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:44.783522 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.783468 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-bound-sa-token\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:44.783567 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.783557 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tghz\" (UniqueName: \"kubernetes.io/projected/e79ec42c-3696-40e7-94de-57aad9a9f92c-kube-api-access-4tghz\") pod \"ingress-canary-9sgpk\" (UID: \"e79ec42c-3696-40e7-94de-57aad9a9f92c\") " pod="openshift-ingress-canary/ingress-canary-9sgpk" Nov 02 01:24:44.788618 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.788586 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-862mp\" (UniqueName: \"kubernetes.io/projected/91cb4fd5-ed27-4a79-b172-be80b203bf0f-kube-api-access-862mp\") pod \"insights-operator-7c4986d449-tgsq4\" (UID: \"91cb4fd5-ed27-4a79-b172-be80b203bf0f\") " pod="openshift-insights/insights-operator-7c4986d449-tgsq4" Nov 02 01:24:44.789076 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.789052 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq7hj\" (UniqueName: \"kubernetes.io/projected/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-kube-api-access-rq7hj\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:44.789191 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.789093 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh2r4\" (UniqueName: \"kubernetes.io/projected/d628518e-2ab7-4acb-acf7-6fc96f359a64-kube-api-access-rh2r4\") pod \"network-metrics-daemon-flwvx\" (UID: \"d628518e-2ab7-4acb-acf7-6fc96f359a64\") " pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:44.789191 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.789125 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-724nj\" (UniqueName: \"kubernetes.io/projected/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-kube-api-access-724nj\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:44.789191 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.789154 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4h6s\" (UniqueName: \"kubernetes.io/projected/028b56de-c4b9-47c1-8bfa-761877176429-kube-api-access-k4h6s\") pod \"service-ca-5fffc97bdf-qkqj6\" (UID: \"028b56de-c4b9-47c1-8bfa-761877176429\") " pod="openshift-service-ca/service-ca-5fffc97bdf-qkqj6" Nov 02 01:24:44.789191 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.789170 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjz2b\" (UniqueName: \"kubernetes.io/projected/8618e1e8-4d71-4f7a-b2a1-1b950190f110-kube-api-access-wjz2b\") pod \"kas-connection-checker-685895c7c9-xmlts\" (UID: \"8618e1e8-4d71-4f7a-b2a1-1b950190f110\") " pod="kube-system/kas-connection-checker-685895c7c9-xmlts" Nov 02 01:24:44.789191 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.789158 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvshf\" (UniqueName: \"kubernetes.io/projected/2c8cbfce-aaf5-4362-930e-b15539b2dbc4-kube-api-access-jvshf\") pod \"network-check-source-6f78d59f99-58fr4\" (UID: \"2c8cbfce-aaf5-4362-930e-b15539b2dbc4\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-58fr4" Nov 02 01:24:44.789454 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.789205 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqppx\" (UniqueName: \"kubernetes.io/projected/9b91ad19-9236-4395-9e54-09470929bf35-kube-api-access-lqppx\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:44.789598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.789570 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpwzf\" (UniqueName: \"kubernetes.io/projected/4abd5311-4a92-4ac5-8aa0-779dd8774577-kube-api-access-tpwzf\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jtll9\" (UID: \"4abd5311-4a92-4ac5-8aa0-779dd8774577\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" Nov 02 01:24:44.789598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.789588 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4wlx\" (UniqueName: \"kubernetes.io/projected/a1ed8a22-43d0-4ca3-9f7b-900231ad3685-kube-api-access-b4wlx\") pod \"node-resolver-ht26h\" (UID: \"a1ed8a22-43d0-4ca3-9f7b-900231ad3685\") " pod="openshift-dns/node-resolver-ht26h" Nov 02 01:24:44.789850 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.789819 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s6ct\" (UniqueName: \"kubernetes.io/projected/35ae4588-f166-4563-aae7-b34b5b053d0f-kube-api-access-6s6ct\") pod \"service-ca-operator-544d749c8c-868jk\" (UID: \"35ae4588-f166-4563-aae7-b34b5b053d0f\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" Nov 02 01:24:44.789904 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.789844 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrnrh\" (UniqueName: \"kubernetes.io/projected/673cb927-a5fc-4237-9b4c-fb196801d70c-kube-api-access-xrnrh\") pod \"iptables-alerter-zzm69\" (UID: \"673cb927-a5fc-4237-9b4c-fb196801d70c\") " pod="openshift-network-operator/iptables-alerter-zzm69" Nov 02 01:24:44.789960 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.789918 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zjgc\" (UniqueName: \"kubernetes.io/projected/e534da40-0733-4670-ba7a-839e0e4b1b4f-kube-api-access-5zjgc\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.790035 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.790019 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnz7p\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-kube-api-access-tnz7p\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:44.790068 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.790043 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh798\" (UniqueName: \"kubernetes.io/projected/eb8ad2e6-7e4f-4c0a-9ea3-99ed98344ba6-kube-api-access-xh798\") pod \"kas-connection-checker-685895c7c9-v8sdv\" (UID: \"eb8ad2e6-7e4f-4c0a-9ea3-99ed98344ba6\") " pod="kube-system/kas-connection-checker-685895c7c9-v8sdv" Nov 02 01:24:44.790312 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.790295 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dzj7\" (UniqueName: \"kubernetes.io/projected/85784876-9397-4bec-8ade-878a15c09f40-kube-api-access-4dzj7\") pod \"network-check-target-kgbsx\" (UID: \"85784876-9397-4bec-8ade-878a15c09f40\") " pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:44.791463 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.791445 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4q2q\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-kube-api-access-r4q2q\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:44.791824 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.791806 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8wfd\" (UniqueName: \"kubernetes.io/projected/7c285da3-2e11-498d-94f1-76c0ce604553-kube-api-access-v8wfd\") pod \"console-operator-7cd56b8dbc-frfx9\" (UID: \"7c285da3-2e11-498d-94f1-76c0ce604553\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:44.818450 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818425 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-sysctl-conf\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.818537 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818453 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-system-cni-dir\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.818537 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818477 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-run-systemd\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.818537 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818492 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-sysctl-d\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.818537 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818528 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs\") pod \"network-metrics-daemon-flwvx\" (UID: \"d628518e-2ab7-4acb-acf7-6fc96f359a64\") " pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:44.818771 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818559 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-run-systemd\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.818771 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818564 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-var-lib-kubelet\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.818771 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818608 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-sysctl-d\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.818771 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818612 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-node-log\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.818771 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818612 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-sysctl-conf\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.818771 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818610 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-var-lib-kubelet\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.818771 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:44.818639 5044 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 01:24:44.818771 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818605 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-system-cni-dir\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.818771 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818694 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-run-ovn-kubernetes\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.818771 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818641 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-run-ovn-kubernetes\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.818771 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818721 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-node-log\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.818771 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818728 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-registration-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:44.818771 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:44.818745 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs podName:d628518e-2ab7-4acb-acf7-6fc96f359a64 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:45.318729087 +0000 UTC m=+1.989981763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs") pod "network-metrics-daemon-flwvx" (UID: "d628518e-2ab7-4acb-acf7-6fc96f359a64") : secret "metrics-daemon-secret" not found Nov 02 01:24:44.818771 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818761 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-registration-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818783 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818803 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-run-multus-certs\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818819 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-run-ovn\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818848 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ntpqz\" (UID: \"fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818865 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-run-multus-certs\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818872 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-socket-dir-parent\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:44.818900 5044 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818906 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-etc-selinux\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818919 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-socket-dir-parent\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818931 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-run-ovn\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818932 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54db4b99-9007-429a-a267-b4924a8729e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:44.818962 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls podName:54db4b99-9007-429a-a267-b4924a8729e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:45.318947134 +0000 UTC m=+1.990199801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2hxq" (UID: "54db4b99-9007-429a-a267-b4924a8729e6") : secret "insights-runtime-extractor-tls" not found Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818982 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-kubelet\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.818981 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-etc-selinux\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819002 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-run\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819026 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b91ad19-9236-4395-9e54-09470929bf35-cnibin\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:44.819035 5044 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819037 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-kubelet\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819052 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b91ad19-9236-4395-9e54-09470929bf35-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819084 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-run\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:44.819101 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert podName:fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:45.319068055 +0000 UTC m=+1.990320736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ntpqz" (UID: "fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41") : secret "networking-console-plugin-cert" not found Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819108 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b91ad19-9236-4395-9e54-09470929bf35-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819116 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-sys\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819110 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b91ad19-9236-4395-9e54-09470929bf35-cnibin\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819135 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-var-lib-openvswitch\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819153 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a128d9e6-7be3-4c73-88f3-ba6b9ea59971-host\") pod \"node-ca-rrf9f\" (UID: \"a128d9e6-7be3-4c73-88f3-ba6b9ea59971\") " pod="openshift-image-registry/node-ca-rrf9f" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819165 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-sys\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819175 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79f97db3-814b-4849-ab8d-305dbcd3449c-dbus\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819181 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-var-lib-openvswitch\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819194 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-socket-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819204 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a128d9e6-7be3-4c73-88f3-ba6b9ea59971-host\") pod \"node-ca-rrf9f\" (UID: \"a128d9e6-7be3-4c73-88f3-ba6b9ea59971\") " pod="openshift-image-registry/node-ca-rrf9f" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819216 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-run-netns\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819241 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79f97db3-814b-4849-ab8d-305dbcd3449c-dbus\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819254 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-metrics-tls\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819258 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-socket-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819273 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-cni-dir\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819287 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-run-netns\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819302 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819321 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-var-lib-cni-bin\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819339 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-hostroot\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819341 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-cni-dir\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819357 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-slash\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819387 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-run-netns\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819394 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54db4b99-9007-429a-a267-b4924a8729e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819403 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-var-lib-cni-bin\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819407 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.819404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819410 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-etc-openvswitch\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819447 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-hostroot\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819450 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-sys-fs\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819456 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-run-netns\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819467 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-slash\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819483 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-cnibin\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819490 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-sys-fs\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819501 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-etc-openvswitch\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819513 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-cni-bin\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819540 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1ed8a22-43d0-4ca3-9f7b-900231ad3685-hosts-file\") pod \"node-resolver-ht26h\" (UID: \"a1ed8a22-43d0-4ca3-9f7b-900231ad3685\") " pod="openshift-dns/node-resolver-ht26h" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819551 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-cnibin\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819567 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819558 5044 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819613 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1ed8a22-43d0-4ca3-9f7b-900231ad3685-hosts-file\") pod \"node-resolver-ht26h\" (UID: \"a1ed8a22-43d0-4ca3-9f7b-900231ad3685\") " pod="openshift-dns/node-resolver-ht26h" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:44.819628 5044 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819560 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-cni-bin\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:44.819693 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls podName:b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b nodeName:}" failed. No retries permitted until 2026-11-02 01:24:45.31965914 +0000 UTC m=+1.990911802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-brlzq" (UID: "b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b") : secret "samples-operator-tls" not found Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819736 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b91ad19-9236-4395-9e54-09470929bf35-system-cni-dir\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819709 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b91ad19-9236-4395-9e54-09470929bf35-system-cni-dir\") pod \"multus-additional-cni-plugins-mgjgw\" (UID: \"9b91ad19-9236-4395-9e54-09470929bf35\") " pod="openshift-multus/multus-additional-cni-plugins-mgjgw" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819781 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-kubernetes\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819800 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/673cb927-a5fc-4237-9b4c-fb196801d70c-host-slash\") pod \"iptables-alerter-zzm69\" (UID: \"673cb927-a5fc-4237-9b4c-fb196801d70c\") " pod="openshift-network-operator/iptables-alerter-zzm69" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819826 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-log-socket\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819846 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819857 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-kubernetes\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819874 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/673cb927-a5fc-4237-9b4c-fb196801d70c-host-slash\") pod \"iptables-alerter-zzm69\" (UID: \"673cb927-a5fc-4237-9b4c-fb196801d70c\") " pod="openshift-network-operator/iptables-alerter-zzm69" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819892 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-log-socket\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819921 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79f97db3-814b-4849-ab8d-305dbcd3449c-kubelet-config\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:44.819940 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle podName:bdebb5b6-76a2-48c8-810d-6f80549a4f3c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:45.31992742 +0000 UTC m=+1.991180107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle") pod "router-default-6cb67dcd4c-zk2vs" (UID: "bdebb5b6-76a2-48c8-810d-6f80549a4f3c") : configmap references non-existent config key: service-ca.crt Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819964 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79f97db3-814b-4849-ab8d-305dbcd3449c-kubelet-config\") pod \"global-pull-secret-syncer-7szx6\" (UID: \"79f97db3-814b-4849-ab8d-305dbcd3449c\") " pod="kube-system/global-pull-secret-syncer-7szx6" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.819966 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820001 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-systemd-units\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820003 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820026 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-cni-netd\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820033 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-systemd-units\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820053 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6sdmz\" (UID: \"59ccadf3-202d-481d-8c8b-c4fa86c92b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820052 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-host-cni-netd\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820073 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-systemd\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820087 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-host\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820137 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-systemd\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820177 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-var-lib-cni-multus\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820183 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-host\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820199 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820214 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-lib-modules\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820229 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820222 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-var-lib-cni-multus\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820267 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-var-lib-kubelet\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:44.820309 5044 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820310 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-var-lib-kubelet\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820340 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-etc-kubernetes\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:44.820364 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs podName:bdebb5b6-76a2-48c8-810d-6f80549a4f3c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:45.320349693 +0000 UTC m=+1.991602366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs") pod "router-default-6cb67dcd4c-zk2vs" (UID: "bdebb5b6-76a2-48c8-810d-6f80549a4f3c") : secret "router-metrics-certs-default" not found Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820373 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-etc-kubernetes\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820382 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-run-openvswitch\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820410 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-lib-modules\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820419 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54db4b99-9007-429a-a267-b4924a8729e6-crio-socket\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820443 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b083c3f1-9c74-4313-a939-ff8aa5769c8d-run-openvswitch\") pod \"ovnkube-node-qzhvs\" (UID: \"b083c3f1-9c74-4313-a939-ff8aa5769c8d\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820450 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-modprobe-d\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820479 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-run-k8s-cni-cncf-io\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820488 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54db4b99-9007-429a-a267-b4924a8729e6-crio-socket\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820509 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-sysconfig\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820523 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-modprobe-d\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820531 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-host-run-k8s-cni-cncf-io\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820538 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-device-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820559 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-conf-dir\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820570 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e534da40-0733-4670-ba7a-839e0e4b1b4f-etc-sysconfig\") pod \"tuned-gwkc5\" (UID: \"e534da40-0733-4670-ba7a-839e0e4b1b4f\") " pod="openshift-cluster-node-tuning-operator/tuned-gwkc5" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820585 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820615 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/25437e9b-ca22-4103-ac9e-c5823f2bbbaf-device-dir\") pod \"aws-ebs-csi-driver-node-lpfsl\" (UID: \"25437e9b-ca22-4103-ac9e-c5823f2bbbaf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lpfsl" Nov 02 01:24:44.821589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.820635 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99-multus-conf-dir\") pod \"multus-wzzsb\" (UID: \"5acf76f4-0fb1-49c1-ad23-dd4ab5f3db99\") " pod="openshift-multus/multus-wzzsb" Nov 02 01:24:44.824610 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.822120 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da5a0b9c-57f5-4741-94f3-fb9fe7b7e699-metrics-tls\") pod \"dns-default-qmrv6\" (UID: \"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699\") " pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:44.824610 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.822589 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls\") pod \"image-registry-69469b6ffb-qmvll\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:44.824610 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.822748 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/59ccadf3-202d-481d-8c8b-c4fa86c92b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6sdmz\" (UID: \"59ccadf3-202d-481d-8c8b-c4fa86c92b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:44.824610 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.822997 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls\") pod \"image-registry-5954c6588f-7h92t\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:44.851990 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.851965 5044 scope.go:122] "RemoveContainer" containerID="78b5c1012635d39793a7b978cf50e9388d5edea5afd509bfce4fa68c48bf2e5a" Nov 02 01:24:44.852230 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.852213 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:44.852888 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.852869 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" Nov 02 01:24:44.863723 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.863704 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:44.863981 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.863954 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:44.878975 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.874385 5044 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:24:44.927107 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:44.927071 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:45.034644 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.031976 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rrf9f_a128d9e6-7be3-4c73-88f3-ba6b9ea59971/node-ca/0.log" Nov 02 01:24:45.115006 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.114975 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:45.257563 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.257485 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:45.287971 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.287935 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69469b6ffb-qmvll"] Nov 02 01:24:45.292534 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.292519 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5954c6588f-7h92t"] Nov 02 01:24:45.327641 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.327619 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:45.327743 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.327653 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:45.327803 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:45.327788 5044 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 01:24:45.327857 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.327800 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:45.327907 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:45.327855 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls podName:b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b nodeName:}" failed. No retries permitted until 2026-11-02 01:24:46.327834852 +0000 UTC m=+2.999087532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-brlzq" (UID: "b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b") : secret "samples-operator-tls" not found Nov 02 01:24:45.327962 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:45.327904 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle podName:bdebb5b6-76a2-48c8-810d-6f80549a4f3c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:46.327892474 +0000 UTC m=+2.999145134 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle") pod "router-default-6cb67dcd4c-zk2vs" (UID: "bdebb5b6-76a2-48c8-810d-6f80549a4f3c") : configmap references non-existent config key: service-ca.crt Nov 02 01:24:45.327962 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:45.327911 5044 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 01:24:45.327962 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.327945 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs\") pod \"network-metrics-daemon-flwvx\" (UID: \"d628518e-2ab7-4acb-acf7-6fc96f359a64\") " pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:45.328086 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:45.327963 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs podName:bdebb5b6-76a2-48c8-810d-6f80549a4f3c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:46.327947497 +0000 UTC m=+2.999200174 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs") pod "router-default-6cb67dcd4c-zk2vs" (UID: "bdebb5b6-76a2-48c8-810d-6f80549a4f3c") : secret "router-metrics-certs-default" not found Nov 02 01:24:45.328086 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:45.327994 5044 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 01:24:45.328086 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.328007 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:45.328086 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:45.328021 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs podName:d628518e-2ab7-4acb-acf7-6fc96f359a64 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:46.328014 +0000 UTC m=+2.999266659 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs") pod "network-metrics-daemon-flwvx" (UID: "d628518e-2ab7-4acb-acf7-6fc96f359a64") : secret "metrics-daemon-secret" not found Nov 02 01:24:45.328086 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.328042 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ntpqz\" (UID: \"fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:45.328314 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:45.328096 5044 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:24:45.328314 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:45.328119 5044 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 01:24:45.328314 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:45.328142 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls podName:54db4b99-9007-429a-a267-b4924a8729e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:46.328130674 +0000 UTC m=+2.999383332 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2hxq" (UID: "54db4b99-9007-429a-a267-b4924a8729e6") : secret "insights-runtime-extractor-tls" not found Nov 02 01:24:45.328314 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:45.328160 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert podName:fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:46.328151995 +0000 UTC m=+2.999404658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ntpqz" (UID: "fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41") : secret "networking-console-plugin-cert" not found Nov 02 01:24:45.328492 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:45.328472 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c07387_ae45_48d3_8b9d_8f328bdfd63c.slice/crio-c6990b114f70582d8499d4cb29e9194eed37fa4675fb3534e78c35a9ffb9dbef WatchSource:0}: Error finding container c6990b114f70582d8499d4cb29e9194eed37fa4675fb3534e78c35a9ffb9dbef: Status 404 returned error can't find the container with id c6990b114f70582d8499d4cb29e9194eed37fa4675fb3534e78c35a9ffb9dbef Nov 02 01:24:45.336882 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.336861 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz"] Nov 02 01:24:45.339815 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.339798 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qmrv6"] Nov 02 01:24:45.416448 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.416423 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:24:45.428807 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:45.428781 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2b846b_5c8b_4982_abc9_809fdbffe82d.slice/crio-d3d3fadca910e29acbad964ec2138091085208cf26e0799faac4c840a6ec9297 WatchSource:0}: Error finding container d3d3fadca910e29acbad964ec2138091085208cf26e0799faac4c840a6ec9297: Status 404 returned error can't find the container with id d3d3fadca910e29acbad964ec2138091085208cf26e0799faac4c840a6ec9297 Nov 02 01:24:45.432648 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.432630 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9sgpk_e79ec42c-3696-40e7-94de-57aad9a9f92c/serve-healthcheck-canary/0.log" Nov 02 01:24:45.477319 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:45.477290 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ccadf3_202d_481d_8c8b_c4fa86c92b8b.slice/crio-8f7893c807fb61aaff33842b34d33b054747a4fa131f470f75ae30f91011d7f3 WatchSource:0}: Error finding container 8f7893c807fb61aaff33842b34d33b054747a4fa131f470f75ae30f91011d7f3: Status 404 returned error can't find the container with id 8f7893c807fb61aaff33842b34d33b054747a4fa131f470f75ae30f91011d7f3 Nov 02 01:24:45.479004 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:45.478968 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5a0b9c_57f5_4741_94f3_fb9fe7b7e699.slice/crio-d84bb389bce633bb0eac43aa10bd0202c1bfe88de21b34d82fa43c12990fb777 WatchSource:0}: Error finding container d84bb389bce633bb0eac43aa10bd0202c1bfe88de21b34d82fa43c12990fb777: Status 404 returned error can't find the container with id d84bb389bce633bb0eac43aa10bd0202c1bfe88de21b34d82fa43c12990fb777 Nov 02 01:24:45.509255 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.509220 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" event={"ID":"d5c07387-ae45-48d3-8b9d-8f328bdfd63c","Type":"ContainerStarted","Data":"ac97acf85f068fecb8405fa16c4e5ad98cf2b3ba278f95bb9c4640c3ff532ab9"} Nov 02 01:24:45.509421 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.509301 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" event={"ID":"d5c07387-ae45-48d3-8b9d-8f328bdfd63c","Type":"ContainerStarted","Data":"c6990b114f70582d8499d4cb29e9194eed37fa4675fb3534e78c35a9ffb9dbef"} Nov 02 01:24:45.511196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.511172 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5954c6588f-7h92t" event={"ID":"7c2b846b-5c8b-4982-abc9-809fdbffe82d","Type":"ContainerStarted","Data":"cf4525032b5a0105ef576e194be04721963bacb1e0a3a29744224572413d7fb4"} Nov 02 01:24:45.511311 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.511213 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5954c6588f-7h92t" event={"ID":"7c2b846b-5c8b-4982-abc9-809fdbffe82d","Type":"ContainerStarted","Data":"d3d3fadca910e29acbad964ec2138091085208cf26e0799faac4c840a6ec9297"} Nov 02 01:24:45.513045 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.513018 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 01:24:45.513460 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.513442 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/1.log" Nov 02 01:24:45.513534 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.513477 5044 generic.go:356] "Generic (PLEG): container finished" podID="7c285da3-2e11-498d-94f1-76c0ce604553" containerID="62caa1aa02cd620020f2ed36f38ef3f0b6c9126e1e1c56edfde8d72fed4fa428" exitCode=255 Nov 02 01:24:45.513593 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.513544 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" event={"ID":"7c285da3-2e11-498d-94f1-76c0ce604553","Type":"ContainerDied","Data":"62caa1aa02cd620020f2ed36f38ef3f0b6c9126e1e1c56edfde8d72fed4fa428"} Nov 02 01:24:45.513650 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.513594 5044 scope.go:122] "RemoveContainer" containerID="78b5c1012635d39793a7b978cf50e9388d5edea5afd509bfce4fa68c48bf2e5a" Nov 02 01:24:45.513911 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.513895 5044 scope.go:122] "RemoveContainer" containerID="62caa1aa02cd620020f2ed36f38ef3f0b6c9126e1e1c56edfde8d72fed4fa428" Nov 02 01:24:45.514086 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:45.514069 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-frfx9_openshift-console-operator(7c285da3-2e11-498d-94f1-76c0ce604553)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" podUID="7c285da3-2e11-498d-94f1-76c0ce604553" Nov 02 01:24:45.514940 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.514919 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qmrv6" event={"ID":"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699","Type":"ContainerStarted","Data":"d84bb389bce633bb0eac43aa10bd0202c1bfe88de21b34d82fa43c12990fb777"} Nov 02 01:24:45.516272 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.516252 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" event={"ID":"59ccadf3-202d-481d-8c8b-c4fa86c92b8b","Type":"ContainerStarted","Data":"8f7893c807fb61aaff33842b34d33b054747a4fa131f470f75ae30f91011d7f3"} Nov 02 01:24:45.902083 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:45.902043 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:46.031700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:46.031593 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-jtll9_4abd5311-4a92-4ac5-8aa0-779dd8774577/kube-storage-version-migrator-operator/0.log" Nov 02 01:24:46.248204 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:46.248171 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:46.338054 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:46.337957 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs\") pod \"network-metrics-daemon-flwvx\" (UID: \"d628518e-2ab7-4acb-acf7-6fc96f359a64\") " pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:46.338054 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:46.338000 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:46.338054 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:46.338027 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ntpqz\" (UID: \"fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:46.338545 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:46.338159 5044 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 01:24:46.338545 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:46.338224 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs podName:d628518e-2ab7-4acb-acf7-6fc96f359a64 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:48.338206199 +0000 UTC m=+5.009458866 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs") pod "network-metrics-daemon-flwvx" (UID: "d628518e-2ab7-4acb-acf7-6fc96f359a64") : secret "metrics-daemon-secret" not found Nov 02 01:24:46.338545 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:46.338276 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:46.338545 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:46.338313 5044 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:24:46.338545 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:46.338314 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:46.338545 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:46.338351 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls podName:54db4b99-9007-429a-a267-b4924a8729e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:48.338339335 +0000 UTC m=+5.009592008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2hxq" (UID: "54db4b99-9007-429a-a267-b4924a8729e6") : secret "insights-runtime-extractor-tls" not found Nov 02 01:24:46.338545 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:46.338354 5044 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 01:24:46.338545 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:46.338374 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:46.338545 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:46.338394 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert podName:fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:48.338382843 +0000 UTC m=+5.009635517 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ntpqz" (UID: "fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41") : secret "networking-console-plugin-cert" not found Nov 02 01:24:46.338545 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:46.338418 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle podName:bdebb5b6-76a2-48c8-810d-6f80549a4f3c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:48.338402727 +0000 UTC m=+5.009655385 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle") pod "router-default-6cb67dcd4c-zk2vs" (UID: "bdebb5b6-76a2-48c8-810d-6f80549a4f3c") : configmap references non-existent config key: service-ca.crt Nov 02 01:24:46.338545 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:46.338422 5044 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 01:24:46.338545 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:46.338426 5044 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 01:24:46.338545 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:46.338480 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls podName:b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b nodeName:}" failed. No retries permitted until 2026-11-02 01:24:48.338464753 +0000 UTC m=+5.009717437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-brlzq" (UID: "b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b") : secret "samples-operator-tls" not found Nov 02 01:24:46.338545 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:46.338499 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs podName:bdebb5b6-76a2-48c8-810d-6f80549a4f3c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:48.33848985 +0000 UTC m=+5.009742514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs") pod "router-default-6cb67dcd4c-zk2vs" (UID: "bdebb5b6-76a2-48c8-810d-6f80549a4f3c") : secret "router-metrics-certs-default" not found Nov 02 01:24:46.374997 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:46.374977 5044 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:46.465798 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:46.465772 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:46.523036 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:46.523001 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-q29rd" event={"ID":"e6c17513-0478-4a05-94dd-0cdad5facf06","Type":"ContainerStarted","Data":"2437fae69c34ac3ab7f3b7102795ccf68e573e799e15a7293f7a45ebd7cdaba3"} Nov 02 01:24:46.524900 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:46.524876 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n5ppw" event={"ID":"476d365d-dd0a-41e6-a186-5e92551be763","Type":"ContainerStarted","Data":"3d4071bfcb744bdc56c24101dc9e6980be28e360d6a857d758ad228c262a1dc8"} Nov 02 01:24:46.527242 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:46.527226 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 01:24:46.527380 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:46.527343 5044 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 01:24:46.527590 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:46.527574 5044 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 01:24:46.527921 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:46.527903 5044 scope.go:122] "RemoveContainer" containerID="62caa1aa02cd620020f2ed36f38ef3f0b6c9126e1e1c56edfde8d72fed4fa428" Nov 02 01:24:46.528091 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:46.528067 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-frfx9_openshift-console-operator(7c285da3-2e11-498d-94f1-76c0ce604553)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" podUID="7c285da3-2e11-498d-94f1-76c0ce604553" Nov 02 01:24:46.610805 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:46.610722 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:46.750985 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:46.750907 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:46.893724 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:46.893589 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:47.374224 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:47.374187 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:24:48.048347 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.048319 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:48.253272 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.253243 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:48.356787 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.356698 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:48.356787 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.356756 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ntpqz\" (UID: \"fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:48.357018 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.356817 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:48.357018 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:48.356848 5044 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:24:48.357018 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:48.356918 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls podName:54db4b99-9007-429a-a267-b4924a8729e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:52.356899685 +0000 UTC m=+9.028152347 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2hxq" (UID: "54db4b99-9007-429a-a267-b4924a8729e6") : secret "insights-runtime-extractor-tls" not found Nov 02 01:24:48.357018 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:48.356918 5044 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 01:24:48.357018 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.356852 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:48.357018 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:48.356980 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert podName:fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41 nodeName:}" failed. No retries permitted until 2026-11-02 01:24:52.356962945 +0000 UTC m=+9.028215604 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-ntpqz" (UID: "fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41") : secret "networking-console-plugin-cert" not found Nov 02 01:24:48.357018 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.357004 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:48.357382 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.357088 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs\") pod \"network-metrics-daemon-flwvx\" (UID: \"d628518e-2ab7-4acb-acf7-6fc96f359a64\") " pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:48.357382 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:48.357117 5044 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 01:24:48.357382 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:48.357177 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs podName:bdebb5b6-76a2-48c8-810d-6f80549a4f3c nodeName:}" failed. No retries permitted until 2026-11-02 01:24:52.35716209 +0000 UTC m=+9.028414767 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs") pod "router-default-6cb67dcd4c-zk2vs" (UID: "bdebb5b6-76a2-48c8-810d-6f80549a4f3c") : secret "router-metrics-certs-default" not found Nov 02 01:24:48.357568 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.357548 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-service-ca-bundle\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:48.359411 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.359382 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-brlzq\" (UID: \"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:48.359757 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.359736 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d628518e-2ab7-4acb-acf7-6fc96f359a64-metrics-certs\") pod \"network-metrics-daemon-flwvx\" (UID: \"d628518e-2ab7-4acb-acf7-6fc96f359a64\") " pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:48.451471 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.451436 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" Nov 02 01:24:48.455073 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.455055 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-flwvx" Nov 02 01:24:48.456201 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.456174 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:48.539689 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.539426 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qmrv6" event={"ID":"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699","Type":"ContainerStarted","Data":"bd74436f4364f5e01176d845fe876c5ce0cd292da158afefa19f0922e5dba93c"} Nov 02 01:24:48.539689 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.539492 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qmrv6" event={"ID":"da5a0b9c-57f5-4741-94f3-fb9fe7b7e699","Type":"ContainerStarted","Data":"e7f13fef34f7255d14eddbd90821f6f32a0adf0ebf9854704e8fc724c779ccf4"} Nov 02 01:24:48.540187 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.540167 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qmrv6" Nov 02 01:24:48.541770 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.541739 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6sdmz" event={"ID":"59ccadf3-202d-481d-8c8b-c4fa86c92b8b","Type":"ContainerStarted","Data":"04ae66e1275b7d491f643235bd89b78406f2bccffa07a359415731f40f765f29"} Nov 02 01:24:48.652986 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.652899 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:48.825925 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.825894 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-flwvx"] Nov 02 01:24:48.848512 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.848485 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:48.866319 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:48.866294 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd628518e_2ab7_4acb_acf7_6fc96f359a64.slice/crio-92403f1a49da72ff506d66991b7739171557eb7d641fad047385dc2960c841f3 WatchSource:0}: Error finding container 92403f1a49da72ff506d66991b7739171557eb7d641fad047385dc2960c841f3: Status 404 returned error can't find the container with id 92403f1a49da72ff506d66991b7739171557eb7d641fad047385dc2960c841f3 Nov 02 01:24:48.880354 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.880330 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq"] Nov 02 01:24:48.956699 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.955413 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:48.989766 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:48.989739 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:49.236111 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:49.236024 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:49.377512 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:49.377464 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:49.522526 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:49.522496 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:49.546664 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:49.546638 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" event={"ID":"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b","Type":"ContainerStarted","Data":"0669211defca467897f0984f8259674b00b3c44b015320a7301498e5aeeeb8db"} Nov 02 01:24:49.547878 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:49.547850 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-flwvx" event={"ID":"d628518e-2ab7-4acb-acf7-6fc96f359a64","Type":"ContainerStarted","Data":"92403f1a49da72ff506d66991b7739171557eb7d641fad047385dc2960c841f3"} Nov 02 01:24:50.018961 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:50.018925 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:50.160530 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:50.160492 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:50.301599 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:50.301499 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:50.449926 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:50.449889 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:50.650003 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:50.649973 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:50.855402 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:50.855321 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:51.557333 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:51.557299 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-flwvx" event={"ID":"d628518e-2ab7-4acb-acf7-6fc96f359a64","Type":"ContainerStarted","Data":"74e722b7eee6d6390c6cea7baae19342c1d61b6835d01e092d1c2a4a8b94cd0b"} Nov 02 01:24:51.557779 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:51.557364 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-flwvx" event={"ID":"d628518e-2ab7-4acb-acf7-6fc96f359a64","Type":"ContainerStarted","Data":"33c5929c82a5258e0b43abfcbf811b9e6f907780886c75f22641dfb9b1bd69c7"} Nov 02 01:24:51.559157 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:51.559133 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" event={"ID":"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b","Type":"ContainerStarted","Data":"de98d0104d6084754f7b1b1fb0ee4f9764c94b17176fa8df6e7366f57f5eb500"} Nov 02 01:24:51.559258 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:51.559180 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-brlzq" event={"ID":"b7d8811a-d4e5-4f0d-9d71-10408d4eeb2b","Type":"ContainerStarted","Data":"d812b68dbe45455ac177064ac5d360977450ee3972d60e23d097182ae38fab95"} Nov 02 01:24:51.697288 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:51.697257 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:51.697610 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:51.697596 5044 scope.go:122] "RemoveContainer" containerID="62caa1aa02cd620020f2ed36f38ef3f0b6c9126e1e1c56edfde8d72fed4fa428" Nov 02 01:24:51.697798 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:51.697779 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-frfx9_openshift-console-operator(7c285da3-2e11-498d-94f1-76c0ce604553)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" podUID="7c285da3-2e11-498d-94f1-76c0ce604553" Nov 02 01:24:52.054744 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:52.054712 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:52.253774 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:52.253738 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:52.390449 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:52.390362 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:24:52.390449 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:52.390419 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:24:52.390449 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:52.390443 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ntpqz\" (UID: \"fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:52.390811 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:52.390538 5044 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 01:24:52.390811 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:52.390575 5044 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 01:24:52.390811 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:52.390607 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs podName:bdebb5b6-76a2-48c8-810d-6f80549a4f3c nodeName:}" failed. No retries permitted until 2026-11-02 01:25:00.390590638 +0000 UTC m=+17.061843309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs") pod "router-default-6cb67dcd4c-zk2vs" (UID: "bdebb5b6-76a2-48c8-810d-6f80549a4f3c") : secret "router-metrics-certs-default" not found Nov 02 01:24:52.390811 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:24:52.390649 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls podName:54db4b99-9007-429a-a267-b4924a8729e6 nodeName:}" failed. No retries permitted until 2026-11-02 01:25:00.390632472 +0000 UTC m=+17.061885130 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2hxq" (UID: "54db4b99-9007-429a-a267-b4924a8729e6") : secret "insights-runtime-extractor-tls" not found Nov 02 01:24:52.392892 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:52.392874 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-ntpqz\" (UID: \"fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:52.447307 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:52.447280 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:52.653948 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:52.653879 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" Nov 02 01:24:52.834430 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:52.834400 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:52.834649 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:52.834632 5044 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 01:24:53.023056 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:53.023029 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-ntpqz"] Nov 02 01:24:53.068065 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:24:53.068034 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef3d6b8_9a4e_4c5d_9f9d_318df3c25e41.slice/crio-b85347a1b2b6d57a43b10fde473e1da2ef6b376ebc3ce3b2582f7bb14687f729 WatchSource:0}: Error finding container b85347a1b2b6d57a43b10fde473e1da2ef6b376ebc3ce3b2582f7bb14687f729: Status 404 returned error can't find the container with id b85347a1b2b6d57a43b10fde473e1da2ef6b376ebc3ce3b2582f7bb14687f729 Nov 02 01:24:53.566851 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:53.566815 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" event={"ID":"fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41","Type":"ContainerStarted","Data":"b85347a1b2b6d57a43b10fde473e1da2ef6b376ebc3ce3b2582f7bb14687f729"} Nov 02 01:24:53.861238 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:53.861170 5044 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-6.us-west-1.compute.internal" event="NodeReady" Nov 02 01:24:54.000323 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:54.000293 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzhvs" Nov 02 01:24:54.112789 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:54.112714 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:24:54.252358 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:54.252320 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:54.438952 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:54.438864 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:54.587166 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:54.587132 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:54.728973 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:54.728904 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:54.867225 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:54.867193 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:55.575097 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:55.575063 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-ntpqz" event={"ID":"fef3d6b8-9a4e-4c5d-9f9d-318df3c25e41","Type":"ContainerStarted","Data":"89aacd1d403dc2bede76e2c3bc71ae34761eb8603bcfa6f25284fab4428f5ad3"} Nov 02 01:24:55.895601 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:55.895523 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:56.038109 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:56.038073 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:56.444761 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:56.444734 5044 scope.go:122] "RemoveContainer" containerID="62caa1aa02cd620020f2ed36f38ef3f0b6c9126e1e1c56edfde8d72fed4fa428" Nov 02 01:24:56.579971 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:56.579948 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 01:24:56.580125 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:56.580070 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" event={"ID":"7c285da3-2e11-498d-94f1-76c0ce604553","Type":"ContainerStarted","Data":"fdb513ae43cd4c76e6450c968273971fc70086b0c1762f063eb099d251493f5d"} Nov 02 01:24:56.580531 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:56.580502 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:56.581274 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:56.581249 5044 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-frfx9 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.16:8443/readyz\": dial tcp 10.132.0.16:8443: connect: connection refused" start-of-body= Nov 02 01:24:56.581358 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:56.581292 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" podUID="7c285da3-2e11-498d-94f1-76c0ce604553" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.16:8443/readyz\": dial tcp 10.132.0.16:8443: connect: connection refused" Nov 02 01:24:57.587019 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:57.586986 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-frfx9" Nov 02 01:24:57.829792 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:57.829750 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:57.982544 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:57.982443 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:58.121396 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:58.121364 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:58.265296 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:58.265268 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:58.420868 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:58.420831 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:58.644239 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:58.644166 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:58.787169 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:58.787138 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:59.121545 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:59.121510 5044 patch_prober.go:29] interesting pod/image-registry-5954c6588f-7h92t container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 01:24:59.121735 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:59.121564 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5954c6588f-7h92t" podUID="7c2b846b-5c8b-4982-abc9-809fdbffe82d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:24:59.149511 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:59.149487 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:59.301851 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:59.301822 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:24:59.553990 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:24:59.553960 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qmrv6" Nov 02 01:25:00.457779 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:00.457742 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:25:00.458362 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:00.457798 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:25:00.460222 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:00.460196 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54db4b99-9007-429a-a267-b4924a8729e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2hxq\" (UID: \"54db4b99-9007-429a-a267-b4924a8729e6\") " pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:25:00.460349 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:00.460330 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdebb5b6-76a2-48c8-810d-6f80549a4f3c-metrics-certs\") pod \"router-default-6cb67dcd4c-zk2vs\" (UID: \"bdebb5b6-76a2-48c8-810d-6f80549a4f3c\") " pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:25:00.465165 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:00.465148 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r2hxq" Nov 02 01:25:00.750303 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:00.750223 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:25:00.835047 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:00.835014 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r2hxq"] Nov 02 01:25:00.835178 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:00.835083 5044 patch_prober.go:29] interesting pod/image-registry-69469b6ffb-qmvll container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 01:25:00.835178 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:00.835137 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" podUID="d5c07387-ae45-48d3-8b9d-8f328bdfd63c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:25:00.837581 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:25:00.837556 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54db4b99_9007_429a_a267_b4924a8729e6.slice/crio-b024464b8059dcd23fdfefd96b342a8f4f20ac757076fd956e91bcf631333305 WatchSource:0}: Error finding container b024464b8059dcd23fdfefd96b342a8f4f20ac757076fd956e91bcf631333305: Status 404 returned error can't find the container with id b024464b8059dcd23fdfefd96b342a8f4f20ac757076fd956e91bcf631333305 Nov 02 01:25:01.121574 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.121471 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6cb67dcd4c-zk2vs"] Nov 02 01:25:01.124059 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:25:01.124034 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdebb5b6_76a2_48c8_810d_6f80549a4f3c.slice/crio-7938c46a2123a970e89f8107a8b2e9df809681b49da95c6d27ac44c760970e1e WatchSource:0}: Error finding container 7938c46a2123a970e89f8107a8b2e9df809681b49da95c6d27ac44c760970e1e: Status 404 returned error can't find the container with id 7938c46a2123a970e89f8107a8b2e9df809681b49da95c6d27ac44c760970e1e Nov 02 01:25:01.424479 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.424407 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69469b6ffb-qmvll"] Nov 02 01:25:01.430406 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.430364 5044 patch_prober.go:29] interesting pod/image-registry-69469b6ffb-qmvll container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 01:25:01.430581 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.430412 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" podUID="d5c07387-ae45-48d3-8b9d-8f328bdfd63c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:25:01.431570 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.431541 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-bd7bl"] Nov 02 01:25:01.434869 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.434847 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kd7nc"] Nov 02 01:25:01.435014 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.434994 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-bd7bl" Nov 02 01:25:01.437912 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.437885 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kd7nc" Nov 02 01:25:01.466663 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.466630 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d75a9f61-49b8-4bf0-b3e5-4200e3ec7332-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-kd7nc\" (UID: \"d75a9f61-49b8-4bf0-b3e5-4200e3ec7332\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kd7nc" Nov 02 01:25:01.467145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.466696 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4skt\" (UniqueName: \"kubernetes.io/projected/772bc38a-1c47-4a45-81a5-5089bd615248-kube-api-access-n4skt\") pod \"downloads-6dd789c8b7-bd7bl\" (UID: \"772bc38a-1c47-4a45-81a5-5089bd615248\") " pod="openshift-console/downloads-6dd789c8b7-bd7bl" Nov 02 01:25:01.467145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.466843 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/772bc38a-1c47-4a45-81a5-5089bd615248-tmp\") pod \"downloads-6dd789c8b7-bd7bl\" (UID: \"772bc38a-1c47-4a45-81a5-5089bd615248\") " pod="openshift-console/downloads-6dd789c8b7-bd7bl" Nov 02 01:25:01.495316 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.495281 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 01:25:01.495472 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.495367 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-kfb6h\"" Nov 02 01:25:01.495472 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.495384 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 01:25:01.507193 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.507162 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kd7nc"] Nov 02 01:25:01.510197 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.510170 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-bd7bl"] Nov 02 01:25:01.524483 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.524457 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75698b548d-m6pkk"] Nov 02 01:25:01.527636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.527621 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.567923 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.567891 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9rrq\" (UniqueName: \"kubernetes.io/projected/2698bb9f-a712-4d25-ae68-bdea717ef9aa-kube-api-access-r9rrq\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.568097 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.567932 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2698bb9f-a712-4d25-ae68-bdea717ef9aa-bound-sa-token\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.568097 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.567967 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d75a9f61-49b8-4bf0-b3e5-4200e3ec7332-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-kd7nc\" (UID: \"d75a9f61-49b8-4bf0-b3e5-4200e3ec7332\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kd7nc" Nov 02 01:25:01.568188 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.568092 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2698bb9f-a712-4d25-ae68-bdea717ef9aa-image-registry-private-configuration\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.568188 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.568123 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4skt\" (UniqueName: \"kubernetes.io/projected/772bc38a-1c47-4a45-81a5-5089bd615248-kube-api-access-n4skt\") pod \"downloads-6dd789c8b7-bd7bl\" (UID: \"772bc38a-1c47-4a45-81a5-5089bd615248\") " pod="openshift-console/downloads-6dd789c8b7-bd7bl" Nov 02 01:25:01.568188 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.568140 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2698bb9f-a712-4d25-ae68-bdea717ef9aa-registry-certificates\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.568342 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.568209 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2698bb9f-a712-4d25-ae68-bdea717ef9aa-ca-trust-extracted\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.568342 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.568245 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2698bb9f-a712-4d25-ae68-bdea717ef9aa-trusted-ca\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.568342 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.568276 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2698bb9f-a712-4d25-ae68-bdea717ef9aa-registry-tls\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.568342 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.568327 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/772bc38a-1c47-4a45-81a5-5089bd615248-tmp\") pod \"downloads-6dd789c8b7-bd7bl\" (UID: \"772bc38a-1c47-4a45-81a5-5089bd615248\") " pod="openshift-console/downloads-6dd789c8b7-bd7bl" Nov 02 01:25:01.568524 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.568381 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2698bb9f-a712-4d25-ae68-bdea717ef9aa-installation-pull-secrets\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.568778 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.568758 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/772bc38a-1c47-4a45-81a5-5089bd615248-tmp\") pod \"downloads-6dd789c8b7-bd7bl\" (UID: \"772bc38a-1c47-4a45-81a5-5089bd615248\") " pod="openshift-console/downloads-6dd789c8b7-bd7bl" Nov 02 01:25:01.596663 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.596633 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 01:25:01.596904 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.596887 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hrf9q\"" Nov 02 01:25:01.597570 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.597541 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r2hxq" event={"ID":"54db4b99-9007-429a-a267-b4924a8729e6","Type":"ContainerStarted","Data":"e5cacac5e4ec8e09a5f94fc1c5ebad59039c338d8709a1892f82b24ce0be3feb"} Nov 02 01:25:01.597695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.597607 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r2hxq" event={"ID":"54db4b99-9007-429a-a267-b4924a8729e6","Type":"ContainerStarted","Data":"b024464b8059dcd23fdfefd96b342a8f4f20ac757076fd956e91bcf631333305"} Nov 02 01:25:01.599258 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.599228 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" event={"ID":"bdebb5b6-76a2-48c8-810d-6f80549a4f3c","Type":"ContainerStarted","Data":"9e753a145505f3821ec780fe703415fd469766bd12f860b529c3d7ca39fa60d2"} Nov 02 01:25:01.599353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.599291 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" event={"ID":"bdebb5b6-76a2-48c8-810d-6f80549a4f3c","Type":"ContainerStarted","Data":"7938c46a2123a970e89f8107a8b2e9df809681b49da95c6d27ac44c760970e1e"} Nov 02 01:25:01.600933 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.600912 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d75a9f61-49b8-4bf0-b3e5-4200e3ec7332-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-kd7nc\" (UID: \"d75a9f61-49b8-4bf0-b3e5-4200e3ec7332\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kd7nc" Nov 02 01:25:01.609936 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.609910 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75698b548d-m6pkk"] Nov 02 01:25:01.630784 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.630752 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4skt\" (UniqueName: \"kubernetes.io/projected/772bc38a-1c47-4a45-81a5-5089bd615248-kube-api-access-n4skt\") pod \"downloads-6dd789c8b7-bd7bl\" (UID: \"772bc38a-1c47-4a45-81a5-5089bd615248\") " pod="openshift-console/downloads-6dd789c8b7-bd7bl" Nov 02 01:25:01.669306 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.669271 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9rrq\" (UniqueName: \"kubernetes.io/projected/2698bb9f-a712-4d25-ae68-bdea717ef9aa-kube-api-access-r9rrq\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.669306 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.669311 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2698bb9f-a712-4d25-ae68-bdea717ef9aa-bound-sa-token\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.669522 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.669339 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2698bb9f-a712-4d25-ae68-bdea717ef9aa-image-registry-private-configuration\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.669522 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.669371 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2698bb9f-a712-4d25-ae68-bdea717ef9aa-registry-certificates\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.669522 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.669395 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2698bb9f-a712-4d25-ae68-bdea717ef9aa-ca-trust-extracted\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.669522 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.669413 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2698bb9f-a712-4d25-ae68-bdea717ef9aa-trusted-ca\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.669522 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.669448 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2698bb9f-a712-4d25-ae68-bdea717ef9aa-registry-tls\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.669971 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.669949 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2698bb9f-a712-4d25-ae68-bdea717ef9aa-ca-trust-extracted\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.670108 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.670081 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2698bb9f-a712-4d25-ae68-bdea717ef9aa-installation-pull-secrets\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.670407 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.670379 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2698bb9f-a712-4d25-ae68-bdea717ef9aa-registry-certificates\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.670562 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.670537 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2698bb9f-a712-4d25-ae68-bdea717ef9aa-trusted-ca\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.672305 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.672280 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2698bb9f-a712-4d25-ae68-bdea717ef9aa-registry-tls\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.672430 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.672408 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2698bb9f-a712-4d25-ae68-bdea717ef9aa-image-registry-private-configuration\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.672846 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.672824 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2698bb9f-a712-4d25-ae68-bdea717ef9aa-installation-pull-secrets\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.713805 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.713713 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:01.733682 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.733625 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2698bb9f-a712-4d25-ae68-bdea717ef9aa-bound-sa-token\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.733844 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.733825 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9rrq\" (UniqueName: \"kubernetes.io/projected/2698bb9f-a712-4d25-ae68-bdea717ef9aa-kube-api-access-r9rrq\") pod \"image-registry-75698b548d-m6pkk\" (UID: \"2698bb9f-a712-4d25-ae68-bdea717ef9aa\") " pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.751197 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.751160 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-bd7bl" Nov 02 01:25:01.751371 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.751206 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:25:01.754914 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.754890 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:25:01.757636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.757620 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kd7nc" Nov 02 01:25:01.840137 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.840112 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:01.947900 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:01.947863 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:02.088615 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:02.088582 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:02.131631 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:02.131601 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-bd7bl"] Nov 02 01:25:02.174271 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:02.174242 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kd7nc"] Nov 02 01:25:02.181399 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:25:02.181363 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772bc38a_1c47_4a45_81a5_5089bd615248.slice/crio-3377e604bd1ded4f87dcfd133467e843ff90a44ef0b4ca77250bd5043888ba68 WatchSource:0}: Error finding container 3377e604bd1ded4f87dcfd133467e843ff90a44ef0b4ca77250bd5043888ba68: Status 404 returned error can't find the container with id 3377e604bd1ded4f87dcfd133467e843ff90a44ef0b4ca77250bd5043888ba68 Nov 02 01:25:02.234744 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:02.234716 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:02.244082 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:25:02.244056 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75a9f61_49b8_4bf0_b3e5_4200e3ec7332.slice/crio-6d4023a03bf4d4e29d1f5c73ca1658a155b0f76901dfdf5d96ff8269c8d5d240 WatchSource:0}: Error finding container 6d4023a03bf4d4e29d1f5c73ca1658a155b0f76901dfdf5d96ff8269c8d5d240: Status 404 returned error can't find the container with id 6d4023a03bf4d4e29d1f5c73ca1658a155b0f76901dfdf5d96ff8269c8d5d240 Nov 02 01:25:02.247596 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:02.247574 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75698b548d-m6pkk"] Nov 02 01:25:02.249554 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:25:02.249528 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2698bb9f_a712_4d25_ae68_bdea717ef9aa.slice/crio-a3b2a290093c23d6e25f06b8c161695a2523946017b9f6d3eafbb5efce7c1db3 WatchSource:0}: Error finding container a3b2a290093c23d6e25f06b8c161695a2523946017b9f6d3eafbb5efce7c1db3: Status 404 returned error can't find the container with id a3b2a290093c23d6e25f06b8c161695a2523946017b9f6d3eafbb5efce7c1db3 Nov 02 01:25:02.391419 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:02.391351 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:02.604392 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:02.604355 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75698b548d-m6pkk" event={"ID":"2698bb9f-a712-4d25-ae68-bdea717ef9aa","Type":"ContainerStarted","Data":"6a1c8b5d3472b4795c14c2e69f280c966e13ec8e1417522abac28c89fd498321"} Nov 02 01:25:02.604872 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:02.604416 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75698b548d-m6pkk" event={"ID":"2698bb9f-a712-4d25-ae68-bdea717ef9aa","Type":"ContainerStarted","Data":"a3b2a290093c23d6e25f06b8c161695a2523946017b9f6d3eafbb5efce7c1db3"} Nov 02 01:25:02.604872 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:02.604486 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:02.605684 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:02.605643 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-bd7bl" event={"ID":"772bc38a-1c47-4a45-81a5-5089bd615248","Type":"ContainerStarted","Data":"3377e604bd1ded4f87dcfd133467e843ff90a44ef0b4ca77250bd5043888ba68"} Nov 02 01:25:02.607022 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:02.606999 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kd7nc" event={"ID":"d75a9f61-49b8-4bf0-b3e5-4200e3ec7332","Type":"ContainerStarted","Data":"6d4023a03bf4d4e29d1f5c73ca1658a155b0f76901dfdf5d96ff8269c8d5d240"} Nov 02 01:25:02.608971 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:02.608935 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r2hxq" event={"ID":"54db4b99-9007-429a-a267-b4924a8729e6","Type":"ContainerStarted","Data":"6956dabe0be1b3a916c9acc97f4af037ae78200361adb9b022977a15db69f018"} Nov 02 01:25:02.609111 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:02.609094 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:25:02.610339 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:02.610310 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6cb67dcd4c-zk2vs" Nov 02 01:25:02.727101 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:02.727005 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75698b548d-m6pkk" podStartSLOduration=1.72699059 podStartE2EDuration="1.72699059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:02.726527283 +0000 UTC m=+19.397779965" watchObservedRunningTime="2026-11-02 01:25:02.72699059 +0000 UTC m=+19.398243262" Nov 02 01:25:04.094536 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:04.094512 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7mk45" Nov 02 01:25:04.159898 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:04.159860 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:04.627067 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:04.626586 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kd7nc" event={"ID":"d75a9f61-49b8-4bf0-b3e5-4200e3ec7332","Type":"ContainerStarted","Data":"6e9613d279cb8b6f85a12c70d862819df5d39ccd50ce7acacf64da1aeeef42e7"} Nov 02 01:25:04.745299 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:04.745247 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kd7nc" podStartSLOduration=1.9765696099999999 podStartE2EDuration="3.745234513s" totalImagesPullingTime="1.768664903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:04.744966656 +0000 UTC m=+21.416219339" watchObservedRunningTime="2026-11-02 01:25:04.745234513 +0000 UTC m=+21.416487191" Nov 02 01:25:05.634693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:05.634301 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r2hxq" event={"ID":"54db4b99-9007-429a-a267-b4924a8729e6","Type":"ContainerStarted","Data":"68069ed56f5fba8be6bec88c7f4a33de853bcdab2aba4a941fe8b22d7cb103c8"} Nov 02 01:25:05.634693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:05.634548 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kd7nc" Nov 02 01:25:05.640005 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:05.639972 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kd7nc" Nov 02 01:25:06.296368 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:06.296323 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:06.441500 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:06.441470 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:06.666125 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:06.662584 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:06.944928 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:06.944843 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:07.538842 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:07.538809 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:25:08.235296 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:08.235254 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:08.400450 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:08.400410 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:10.407849 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:10.407809 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:10.556114 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:10.556079 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:10.929049 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:10.929012 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:11.073095 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:11.073057 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:11.430062 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:11.430037 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:25:12.102511 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:12.102432 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:12.663824 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:12.663793 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:12.894145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:12.894113 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:13.181049 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.181002 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:13.374725 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.374691 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:13.507704 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.507618 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-z9mp6"] Nov 02 01:25:13.536739 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.536705 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:13.538790 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.538765 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.681784 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.681749 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/928ee3de-8fae-4a90-8fea-84b30d084700-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.682243 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.681798 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/928ee3de-8fae-4a90-8fea-84b30d084700-root\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.682243 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.681891 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/928ee3de-8fae-4a90-8fea-84b30d084700-node-exporter-accelerators-collector-config\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.682243 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.681985 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/928ee3de-8fae-4a90-8fea-84b30d084700-metrics-client-ca\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.682243 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.682119 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/928ee3de-8fae-4a90-8fea-84b30d084700-node-exporter-textfile\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.682243 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.682157 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl7gt\" (UniqueName: \"kubernetes.io/projected/928ee3de-8fae-4a90-8fea-84b30d084700-kube-api-access-dl7gt\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.682480 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.682285 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/928ee3de-8fae-4a90-8fea-84b30d084700-sys\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.682480 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.682343 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/928ee3de-8fae-4a90-8fea-84b30d084700-node-exporter-wtmp\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.682480 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.682407 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/928ee3de-8fae-4a90-8fea-84b30d084700-node-exporter-tls\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.710428 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.710390 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 01:25:13.710583 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.710572 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 01:25:13.710728 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.710641 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 01:25:13.710728 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.710705 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4dgbb\"" Nov 02 01:25:13.710863 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.710815 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 01:25:13.720769 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.720742 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:13.783878 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.783829 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/928ee3de-8fae-4a90-8fea-84b30d084700-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.784080 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.783884 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/928ee3de-8fae-4a90-8fea-84b30d084700-root\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.784080 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.783913 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/928ee3de-8fae-4a90-8fea-84b30d084700-node-exporter-accelerators-collector-config\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.784080 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.783952 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/928ee3de-8fae-4a90-8fea-84b30d084700-metrics-client-ca\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.784080 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.783988 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/928ee3de-8fae-4a90-8fea-84b30d084700-node-exporter-textfile\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.784080 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.783993 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/928ee3de-8fae-4a90-8fea-84b30d084700-root\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.784080 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.784018 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl7gt\" (UniqueName: \"kubernetes.io/projected/928ee3de-8fae-4a90-8fea-84b30d084700-kube-api-access-dl7gt\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.784341 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.784093 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/928ee3de-8fae-4a90-8fea-84b30d084700-sys\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.784341 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.784133 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/928ee3de-8fae-4a90-8fea-84b30d084700-node-exporter-wtmp\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.784341 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.784187 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/928ee3de-8fae-4a90-8fea-84b30d084700-node-exporter-tls\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.784474 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.784335 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/928ee3de-8fae-4a90-8fea-84b30d084700-sys\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.784572 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.784556 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/928ee3de-8fae-4a90-8fea-84b30d084700-node-exporter-wtmp\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.785421 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.785391 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/928ee3de-8fae-4a90-8fea-84b30d084700-node-exporter-textfile\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.785534 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.785424 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/928ee3de-8fae-4a90-8fea-84b30d084700-metrics-client-ca\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.785852 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.785826 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/928ee3de-8fae-4a90-8fea-84b30d084700-node-exporter-accelerators-collector-config\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.787078 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.787054 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/928ee3de-8fae-4a90-8fea-84b30d084700-node-exporter-tls\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.789276 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.789252 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/928ee3de-8fae-4a90-8fea-84b30d084700-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.846850 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.846819 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl7gt\" (UniqueName: \"kubernetes.io/projected/928ee3de-8fae-4a90-8fea-84b30d084700-kube-api-access-dl7gt\") pod \"node-exporter-z9mp6\" (UID: \"928ee3de-8fae-4a90-8fea-84b30d084700\") " pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.879840 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.879812 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:13.910093 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:13.910068 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z9mp6" Nov 02 01:25:13.943723 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:25:13.943691 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928ee3de_8fae_4a90_8fea_84b30d084700.slice/crio-ed5cd1a34649ece635e3c99b97e7dc4a2759bf2ec0f302b98d2976018a033de1 WatchSource:0}: Error finding container ed5cd1a34649ece635e3c99b97e7dc4a2759bf2ec0f302b98d2976018a033de1: Status 404 returned error can't find the container with id ed5cd1a34649ece635e3c99b97e7dc4a2759bf2ec0f302b98d2976018a033de1 Nov 02 01:25:14.055453 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:14.055362 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:14.678538 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:14.678499 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z9mp6" event={"ID":"928ee3de-8fae-4a90-8fea-84b30d084700","Type":"ContainerStarted","Data":"ed5cd1a34649ece635e3c99b97e7dc4a2759bf2ec0f302b98d2976018a033de1"} Nov 02 01:25:14.833608 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:14.833565 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:15.109008 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:15.108968 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:15.254773 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:15.254737 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:15.416988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:15.416894 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:15.519041 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:15.519010 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kgbsx" Nov 02 01:25:15.611879 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:15.611842 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:15.823884 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:15.823848 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:16.573167 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:16.573130 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:16.691013 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:16.690981 5044 generic.go:356] "Generic (PLEG): container finished" podID="928ee3de-8fae-4a90-8fea-84b30d084700" containerID="ea9166fef29cae024ce47f65ccaffaf787a149050f60a41152511f84e029bc00" exitCode=0 Nov 02 01:25:16.691013 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:16.691024 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z9mp6" event={"ID":"928ee3de-8fae-4a90-8fea-84b30d084700","Type":"ContainerDied","Data":"ea9166fef29cae024ce47f65ccaffaf787a149050f60a41152511f84e029bc00"} Nov 02 01:25:16.717638 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:16.717614 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:17.546122 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:17.546062 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:17.699342 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:17.699302 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z9mp6" event={"ID":"928ee3de-8fae-4a90-8fea-84b30d084700","Type":"ContainerStarted","Data":"72aa3617b328c5f016a316a9729e538a3734a3f9ffa1b797f17a0e8b8ff84d02"} Nov 02 01:25:17.699836 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:17.699370 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z9mp6" event={"ID":"928ee3de-8fae-4a90-8fea-84b30d084700","Type":"ContainerStarted","Data":"cd29d6ae908ec36b6db57cf3bc46826f5f974cb38dcfd67cd8745471386ec639"} Nov 02 01:25:17.705715 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:17.705666 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:17.822302 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:17.822198 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-z9mp6" podStartSLOduration=2.6666080770000002 podStartE2EDuration="4.822179517s" totalImagesPullingTime="2.060667086s" totalInitContainerRuntime="94.904354ms" isStatefulPod=true podCreationTimestamp="2026-11-02 01:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:17.820415698 +0000 UTC m=+34.491668380" watchObservedRunningTime="2026-11-02 01:25:17.822179517 +0000 UTC m=+34.493432197" Nov 02 01:25:17.905321 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:17.905241 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:18.052577 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:18.052540 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:18.267300 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:18.267267 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:19.711406 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:19.711327 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-bd7bl" event={"ID":"772bc38a-1c47-4a45-81a5-5089bd615248","Type":"ContainerStarted","Data":"d7396e175df24c751bba21f67b9af605b1fe4f26ac241258d128587c58dc4a54"} Nov 02 01:25:19.712361 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:19.712292 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-bd7bl" Nov 02 01:25:19.718230 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:19.718210 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-bd7bl" Nov 02 01:25:19.932906 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:19.932843 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-bd7bl" podStartSLOduration=1.818829254 podStartE2EDuration="18.932823462s" totalImagesPullingTime="17.113994208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:25:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:25:19.881182107 +0000 UTC m=+36.552434788" watchObservedRunningTime="2026-11-02 01:25:19.932823462 +0000 UTC m=+36.604076136" Nov 02 01:25:19.995142 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:19.995056 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:20.147344 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:20.147305 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:22.492271 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:22.492237 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:23.625159 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:23.625131 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75698b548d-m6pkk" Nov 02 01:25:23.864713 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:23.864655 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5954c6588f-7h92t"] Nov 02 01:25:23.999156 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:23.999071 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:24.169388 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:24.169357 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:24.175827 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:25:24.175795 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:25:24.315686 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:24.315639 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:25.927426 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:25.927391 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:26.080204 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.080165 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:26.447741 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.447699 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" podUID="d5c07387-ae45-48d3-8b9d-8f328bdfd63c" containerName="registry" containerID="cri-o://ac97acf85f068fecb8405fa16c4e5ad98cf2b3ba278f95bb9c4640c3ff532ab9" gracePeriod=30 Nov 02 01:25:26.632393 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.632354 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:26.747008 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.746978 5044 generic.go:356] "Generic (PLEG): container finished" podID="d5c07387-ae45-48d3-8b9d-8f328bdfd63c" containerID="ac97acf85f068fecb8405fa16c4e5ad98cf2b3ba278f95bb9c4640c3ff532ab9" exitCode=0 Nov 02 01:25:26.747142 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.747055 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" event={"ID":"d5c07387-ae45-48d3-8b9d-8f328bdfd63c","Type":"ContainerDied","Data":"ac97acf85f068fecb8405fa16c4e5ad98cf2b3ba278f95bb9c4640c3ff532ab9"} Nov 02 01:25:26.747142 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.747109 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" event={"ID":"d5c07387-ae45-48d3-8b9d-8f328bdfd63c","Type":"ContainerDied","Data":"c6990b114f70582d8499d4cb29e9194eed37fa4675fb3534e78c35a9ffb9dbef"} Nov 02 01:25:26.747142 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.747126 5044 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6990b114f70582d8499d4cb29e9194eed37fa4675fb3534e78c35a9ffb9dbef" Nov 02 01:25:26.753949 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.753664 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:25:26.792307 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.792266 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:26.904379 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.904342 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-installation-pull-secrets\") pod \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " Nov 02 01:25:26.904562 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.904410 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-kube-api-access-tnz7p\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-kube-api-access-tnz7p\") pod \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " Nov 02 01:25:26.904562 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.904440 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-image-registry-private-configuration\") pod \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " Nov 02 01:25:26.904562 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.904530 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls\") pod \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " Nov 02 01:25:26.904769 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.904579 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-trusted-ca\") pod \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " Nov 02 01:25:26.904769 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.904614 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-bound-sa-token\") pod \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " Nov 02 01:25:26.904769 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.904707 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-certificates\") pod \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " Nov 02 01:25:26.904931 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.904783 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-ca-trust-extracted\") pod \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\" (UID: \"d5c07387-ae45-48d3-8b9d-8f328bdfd63c\") " Nov 02 01:25:26.905478 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.905444 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-certificates" pod "d5c07387-ae45-48d3-8b9d-8f328bdfd63c" (UID: "d5c07387-ae45-48d3-8b9d-8f328bdfd63c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:25:26.905612 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.905509 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-trusted-ca" pod "d5c07387-ae45-48d3-8b9d-8f328bdfd63c" (UID: "d5c07387-ae45-48d3-8b9d-8f328bdfd63c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:25:26.911990 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.911956 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-bound-sa-token" pod "d5c07387-ae45-48d3-8b9d-8f328bdfd63c" (UID: "d5c07387-ae45-48d3-8b9d-8f328bdfd63c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:25:26.912114 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.912007 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls" pod "d5c07387-ae45-48d3-8b9d-8f328bdfd63c" (UID: "d5c07387-ae45-48d3-8b9d-8f328bdfd63c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:25:26.912114 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.912047 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-installation-pull-secrets" pod "d5c07387-ae45-48d3-8b9d-8f328bdfd63c" (UID: "d5c07387-ae45-48d3-8b9d-8f328bdfd63c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:25:26.912240 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.912134 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-image-registry-private-configuration" pod "d5c07387-ae45-48d3-8b9d-8f328bdfd63c" (UID: "d5c07387-ae45-48d3-8b9d-8f328bdfd63c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:25:26.912240 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.912204 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-kube-api-access-tnz7p" pod "d5c07387-ae45-48d3-8b9d-8f328bdfd63c" (UID: "d5c07387-ae45-48d3-8b9d-8f328bdfd63c"). InnerVolumeSpecName "kube-api-access-tnz7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:25:26.913835 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:26.913808 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-ca-trust-extracted" pod "d5c07387-ae45-48d3-8b9d-8f328bdfd63c" (UID: "d5c07387-ae45-48d3-8b9d-8f328bdfd63c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:25:27.006196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:27.006125 5044 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-installation-pull-secrets\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:25:27.006196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:27.006161 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnz7p\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-kube-api-access-tnz7p\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:25:27.006196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:27.006178 5044 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-image-registry-private-configuration\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:25:27.006196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:27.006192 5044 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-tls\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:25:27.006196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:27.006202 5044 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-trusted-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:25:27.006788 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:27.006214 5044 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-bound-sa-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:25:27.006788 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:27.006229 5044 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-registry-certificates\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:25:27.006788 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:27.006238 5044 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d5c07387-ae45-48d3-8b9d-8f328bdfd63c-ca-trust-extracted\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:25:27.752304 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:27.752233 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69469b6ffb-qmvll" Nov 02 01:25:27.971790 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:27.971749 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69469b6ffb-qmvll"] Nov 02 01:25:28.022561 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:28.022522 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69469b6ffb-qmvll"] Nov 02 01:25:29.170293 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:29.170219 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:29.323176 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:29.323140 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:29.450085 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:29.450001 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c07387-ae45-48d3-8b9d-8f328bdfd63c" path="/var/lib/kubelet/pods/d5c07387-ae45-48d3-8b9d-8f328bdfd63c/volumes" Nov 02 01:25:32.212380 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:32.212350 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:32.362185 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:32.362154 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:32.568590 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:32.568556 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:32.718970 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:32.718940 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:32.865956 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:32.865890 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:33.114116 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:33.114085 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:33.295004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:33.294973 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:33.453004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:33.452973 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:33.685083 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:33.685007 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:33.833774 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:33.833743 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:35.976643 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:35.976608 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:36.126080 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:36.126049 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:48.889991 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:48.889931 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5954c6588f-7h92t" podUID="7c2b846b-5c8b-4982-abc9-809fdbffe82d" containerName="registry" containerID="cri-o://cf4525032b5a0105ef576e194be04721963bacb1e0a3a29744224572413d7fb4" gracePeriod=30 Nov 02 01:25:49.172374 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.172353 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:25:49.292857 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.292820 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c2b846b-5c8b-4982-abc9-809fdbffe82d-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c2b846b-5c8b-4982-abc9-809fdbffe82d-image-registry-private-configuration\") pod \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " Nov 02 01:25:49.293039 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.292880 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c2b846b-5c8b-4982-abc9-809fdbffe82d-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c2b846b-5c8b-4982-abc9-809fdbffe82d-trusted-ca\") pod \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " Nov 02 01:25:49.293039 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.292913 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c2b846b-5c8b-4982-abc9-809fdbffe82d-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c2b846b-5c8b-4982-abc9-809fdbffe82d-ca-trust-extracted\") pod \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " Nov 02 01:25:49.293039 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.292950 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-kube-api-access-r4q2q\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-kube-api-access-r4q2q\") pod \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " Nov 02 01:25:49.293039 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.292980 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-bound-sa-token\") pod \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " Nov 02 01:25:49.293039 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.293014 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls\") pod \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " Nov 02 01:25:49.293344 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.293080 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c2b846b-5c8b-4982-abc9-809fdbffe82d-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c2b846b-5c8b-4982-abc9-809fdbffe82d-installation-pull-secrets\") pod \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " Nov 02 01:25:49.293344 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.293136 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-certificates\") pod \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\" (UID: \"7c2b846b-5c8b-4982-abc9-809fdbffe82d\") " Nov 02 01:25:49.293451 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.293335 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c2b846b-5c8b-4982-abc9-809fdbffe82d-trusted-ca" pod "7c2b846b-5c8b-4982-abc9-809fdbffe82d" (UID: "7c2b846b-5c8b-4982-abc9-809fdbffe82d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:25:49.293770 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.293736 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-certificates" pod "7c2b846b-5c8b-4982-abc9-809fdbffe82d" (UID: "7c2b846b-5c8b-4982-abc9-809fdbffe82d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:25:49.295590 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.295551 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-kube-api-access-r4q2q" pod "7c2b846b-5c8b-4982-abc9-809fdbffe82d" (UID: "7c2b846b-5c8b-4982-abc9-809fdbffe82d"). InnerVolumeSpecName "kube-api-access-r4q2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:25:49.295719 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.295596 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-bound-sa-token" pod "7c2b846b-5c8b-4982-abc9-809fdbffe82d" (UID: "7c2b846b-5c8b-4982-abc9-809fdbffe82d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:25:49.295719 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.295693 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c2b846b-5c8b-4982-abc9-809fdbffe82d-installation-pull-secrets" pod "7c2b846b-5c8b-4982-abc9-809fdbffe82d" (UID: "7c2b846b-5c8b-4982-abc9-809fdbffe82d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:25:49.295825 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.295788 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c2b846b-5c8b-4982-abc9-809fdbffe82d-image-registry-private-configuration" pod "7c2b846b-5c8b-4982-abc9-809fdbffe82d" (UID: "7c2b846b-5c8b-4982-abc9-809fdbffe82d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:25:49.295872 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.295822 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls" pod "7c2b846b-5c8b-4982-abc9-809fdbffe82d" (UID: "7c2b846b-5c8b-4982-abc9-809fdbffe82d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:25:49.300300 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.300254 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c2b846b-5c8b-4982-abc9-809fdbffe82d-ca-trust-extracted" pod "7c2b846b-5c8b-4982-abc9-809fdbffe82d" (UID: "7c2b846b-5c8b-4982-abc9-809fdbffe82d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:25:49.394356 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.394272 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4q2q\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-kube-api-access-r4q2q\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:25:49.394356 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.394298 5044 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-bound-sa-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:25:49.394356 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.394312 5044 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-tls\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:25:49.394356 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.394324 5044 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7c2b846b-5c8b-4982-abc9-809fdbffe82d-installation-pull-secrets\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:25:49.394356 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.394336 5044 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7c2b846b-5c8b-4982-abc9-809fdbffe82d-registry-certificates\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:25:49.394356 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.394358 5044 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7c2b846b-5c8b-4982-abc9-809fdbffe82d-image-registry-private-configuration\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:25:49.394639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.394372 5044 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c2b846b-5c8b-4982-abc9-809fdbffe82d-trusted-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:25:49.394639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.394384 5044 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7c2b846b-5c8b-4982-abc9-809fdbffe82d-ca-trust-extracted\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:25:49.842239 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.842202 5044 generic.go:356] "Generic (PLEG): container finished" podID="7c2b846b-5c8b-4982-abc9-809fdbffe82d" containerID="cf4525032b5a0105ef576e194be04721963bacb1e0a3a29744224572413d7fb4" exitCode=0 Nov 02 01:25:49.842406 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.842249 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5954c6588f-7h92t" Nov 02 01:25:49.842406 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.842286 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5954c6588f-7h92t" event={"ID":"7c2b846b-5c8b-4982-abc9-809fdbffe82d","Type":"ContainerDied","Data":"cf4525032b5a0105ef576e194be04721963bacb1e0a3a29744224572413d7fb4"} Nov 02 01:25:49.842406 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.842337 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5954c6588f-7h92t" event={"ID":"7c2b846b-5c8b-4982-abc9-809fdbffe82d","Type":"ContainerDied","Data":"d3d3fadca910e29acbad964ec2138091085208cf26e0799faac4c840a6ec9297"} Nov 02 01:25:49.842406 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.842367 5044 scope.go:122] "RemoveContainer" containerID="cf4525032b5a0105ef576e194be04721963bacb1e0a3a29744224572413d7fb4" Nov 02 01:25:49.850550 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.850532 5044 scope.go:122] "RemoveContainer" containerID="cf4525032b5a0105ef576e194be04721963bacb1e0a3a29744224572413d7fb4" Nov 02 01:25:49.850851 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:25:49.850832 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf4525032b5a0105ef576e194be04721963bacb1e0a3a29744224572413d7fb4\": container with ID starting with cf4525032b5a0105ef576e194be04721963bacb1e0a3a29744224572413d7fb4 not found: ID does not exist" containerID="cf4525032b5a0105ef576e194be04721963bacb1e0a3a29744224572413d7fb4" Nov 02 01:25:49.850932 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:49.850860 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf4525032b5a0105ef576e194be04721963bacb1e0a3a29744224572413d7fb4"} err="failed to get container status \"cf4525032b5a0105ef576e194be04721963bacb1e0a3a29744224572413d7fb4\": rpc error: code = NotFound desc = could not find container \"cf4525032b5a0105ef576e194be04721963bacb1e0a3a29744224572413d7fb4\": container with ID starting with cf4525032b5a0105ef576e194be04721963bacb1e0a3a29744224572413d7fb4 not found: ID does not exist" Nov 02 01:25:50.013659 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:50.013621 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5954c6588f-7h92t"] Nov 02 01:25:50.064512 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:50.064478 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5954c6588f-7h92t"] Nov 02 01:25:51.450030 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:51.450003 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c2b846b-5c8b-4982-abc9-809fdbffe82d" path="/var/lib/kubelet/pods/7c2b846b-5c8b-4982-abc9-809fdbffe82d/volumes" Nov 02 01:25:54.171296 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:25:54.171257 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:25:55.910663 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:55.910631 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:25:56.061628 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:25:56.061591 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:26:03.901165 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:03.901130 5044 generic.go:356] "Generic (PLEG): container finished" podID="91cb4fd5-ed27-4a79-b172-be80b203bf0f" containerID="b2c932d315e10d88a096b2910463f995c257b4320f82b026d02975419da0db8d" exitCode=0 Nov 02 01:26:03.901545 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:03.901174 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tgsq4" event={"ID":"91cb4fd5-ed27-4a79-b172-be80b203bf0f","Type":"ContainerDied","Data":"b2c932d315e10d88a096b2910463f995c257b4320f82b026d02975419da0db8d"} Nov 02 01:26:03.901545 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:03.901485 5044 scope.go:122] "RemoveContainer" containerID="b2c932d315e10d88a096b2910463f995c257b4320f82b026d02975419da0db8d" Nov 02 01:26:04.906800 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:04.906761 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-tgsq4" event={"ID":"91cb4fd5-ed27-4a79-b172-be80b203bf0f","Type":"ContainerStarted","Data":"01a69b46c7b5a5d5e06600874565596559ccb7978472e360570d8a016c27a333"} Nov 02 01:26:05.312353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.312321 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-5bwtl-qgqdr"] Nov 02 01:26:05.312546 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.312524 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" podUID="eb932801-604f-430e-bfe3-8d4c9a1a7ca6" containerName="insights-gathering" containerID="cri-o://d561c8c28f147177b3d69d7c35d73e9821dadf4125eec9721a577287390da5a3" gracePeriod=30 Nov 02 01:26:05.491352 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.491324 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:26:05.631392 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.631320 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-archives-path\") pod \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " Nov 02 01:26:05.631392 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.631359 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-kube-api-access-drbjx\" (UniqueName: \"kubernetes.io/projected/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-kube-api-access-drbjx\") pod \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " Nov 02 01:26:05.631609 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.631416 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-service-ca-bundle\") pod \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " Nov 02 01:26:05.631609 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.631450 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-trusted-ca-bundle\") pod \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\" (UID: \"eb932801-604f-430e-bfe3-8d4c9a1a7ca6\") " Nov 02 01:26:05.631765 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.631651 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-archives-path" pod "eb932801-604f-430e-bfe3-8d4c9a1a7ca6" (UID: "eb932801-604f-430e-bfe3-8d4c9a1a7ca6"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:26:05.631882 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.631860 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-service-ca-bundle" pod "eb932801-604f-430e-bfe3-8d4c9a1a7ca6" (UID: "eb932801-604f-430e-bfe3-8d4c9a1a7ca6"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:26:05.631932 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.631899 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-trusted-ca-bundle" pod "eb932801-604f-430e-bfe3-8d4c9a1a7ca6" (UID: "eb932801-604f-430e-bfe3-8d4c9a1a7ca6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:26:05.633762 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.633736 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-kube-api-access-drbjx" pod "eb932801-604f-430e-bfe3-8d4c9a1a7ca6" (UID: "eb932801-604f-430e-bfe3-8d4c9a1a7ca6"). InnerVolumeSpecName "kube-api-access-drbjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:26:05.732490 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.732458 5044 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-archives-path\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:26:05.732490 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.732490 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drbjx\" (UniqueName: \"kubernetes.io/projected/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-kube-api-access-drbjx\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:26:05.732702 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.732502 5044 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-service-ca-bundle\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:26:05.732702 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.732511 5044 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb932801-604f-430e-bfe3-8d4c9a1a7ca6-trusted-ca-bundle\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:26:05.911918 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.911835 5044 generic.go:356] "Generic (PLEG): container finished" podID="eb932801-604f-430e-bfe3-8d4c9a1a7ca6" containerID="d561c8c28f147177b3d69d7c35d73e9821dadf4125eec9721a577287390da5a3" exitCode=0 Nov 02 01:26:05.911918 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.911875 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" Nov 02 01:26:05.911918 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.911897 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" event={"ID":"eb932801-604f-430e-bfe3-8d4c9a1a7ca6","Type":"ContainerDied","Data":"d561c8c28f147177b3d69d7c35d73e9821dadf4125eec9721a577287390da5a3"} Nov 02 01:26:05.912450 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.911943 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5bwtl-qgqdr" event={"ID":"eb932801-604f-430e-bfe3-8d4c9a1a7ca6","Type":"ContainerDied","Data":"194ec6187c7b572797fd8f78e02cb66a027fb1b3f1f870aa4adda003f033d9e7"} Nov 02 01:26:05.912450 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.911964 5044 scope.go:122] "RemoveContainer" containerID="d561c8c28f147177b3d69d7c35d73e9821dadf4125eec9721a577287390da5a3" Nov 02 01:26:05.923435 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.923407 5044 scope.go:122] "RemoveContainer" containerID="d561c8c28f147177b3d69d7c35d73e9821dadf4125eec9721a577287390da5a3" Nov 02 01:26:05.923966 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:26:05.923934 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d561c8c28f147177b3d69d7c35d73e9821dadf4125eec9721a577287390da5a3\": container with ID starting with d561c8c28f147177b3d69d7c35d73e9821dadf4125eec9721a577287390da5a3 not found: ID does not exist" containerID="d561c8c28f147177b3d69d7c35d73e9821dadf4125eec9721a577287390da5a3" Nov 02 01:26:05.924075 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:05.923977 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d561c8c28f147177b3d69d7c35d73e9821dadf4125eec9721a577287390da5a3"} err="failed to get container status \"d561c8c28f147177b3d69d7c35d73e9821dadf4125eec9721a577287390da5a3\": rpc error: code = NotFound desc = could not find container \"d561c8c28f147177b3d69d7c35d73e9821dadf4125eec9721a577287390da5a3\": container with ID starting with d561c8c28f147177b3d69d7c35d73e9821dadf4125eec9721a577287390da5a3 not found: ID does not exist" Nov 02 01:26:06.087767 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:06.087733 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-5bwtl-qgqdr"] Nov 02 01:26:06.108094 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:06.108062 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-5bwtl-qgqdr"] Nov 02 01:26:07.450310 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:07.450280 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb932801-604f-430e-bfe3-8d4c9a1a7ca6" path="/var/lib/kubelet/pods/eb932801-604f-430e-bfe3-8d4c9a1a7ca6/volumes" Nov 02 01:26:08.926274 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:08.926241 5044 generic.go:356] "Generic (PLEG): container finished" podID="4abd5311-4a92-4ac5-8aa0-779dd8774577" containerID="1040377acf4b69c4a29c3ce7a8e07950d47512ae78348025b1a64d20c84493a8" exitCode=0 Nov 02 01:26:08.926731 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:08.926312 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" event={"ID":"4abd5311-4a92-4ac5-8aa0-779dd8774577","Type":"ContainerDied","Data":"1040377acf4b69c4a29c3ce7a8e07950d47512ae78348025b1a64d20c84493a8"} Nov 02 01:26:08.926731 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:08.926663 5044 scope.go:122] "RemoveContainer" containerID="1040377acf4b69c4a29c3ce7a8e07950d47512ae78348025b1a64d20c84493a8" Nov 02 01:26:09.931871 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:09.931838 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jtll9" event={"ID":"4abd5311-4a92-4ac5-8aa0-779dd8774577","Type":"ContainerStarted","Data":"b8ca36f009c6b5d8f34756c097dada84c3e01d0c016bb559eb96ec6f9a38408c"} Nov 02 01:26:17.084778 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:17.084730 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:26:17.267293 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:17.267258 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:26:18.966842 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:18.966814 5044 generic.go:356] "Generic (PLEG): container finished" podID="35ae4588-f166-4563-aae7-b34b5b053d0f" containerID="9413267b71c2d2bc04d27f9909f37ce17cf05ff74f22be400d8e43e5303adec8" exitCode=0 Nov 02 01:26:18.967185 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:18.966884 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" event={"ID":"35ae4588-f166-4563-aae7-b34b5b053d0f","Type":"ContainerDied","Data":"9413267b71c2d2bc04d27f9909f37ce17cf05ff74f22be400d8e43e5303adec8"} Nov 02 01:26:18.967185 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:18.967178 5044 scope.go:122] "RemoveContainer" containerID="9413267b71c2d2bc04d27f9909f37ce17cf05ff74f22be400d8e43e5303adec8" Nov 02 01:26:19.971899 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:19.971862 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-868jk" event={"ID":"35ae4588-f166-4563-aae7-b34b5b053d0f","Type":"ContainerStarted","Data":"c1ab3e6aac0a2e7d1648f66cf7b2646f02c52a5815fecd7ad4d6dd1adfdfda34"} Nov 02 01:26:24.171309 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:26:24.171272 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:26:27.471098 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:27.471060 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:26:27.622611 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:27.622572 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:26:27.853727 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:27.853693 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:26:28.031504 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:28.031470 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:26:33.465491 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:33.465457 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:26:54.171663 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:26:54.171582 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:26:55.760273 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:55.760237 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:26:55.903180 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:26:55.903149 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:27:24.171320 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:27:24.171283 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:27:33.798349 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:27:33.798317 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:27:34.860975 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:27:34.860922 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:27:35.004281 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:27:35.004250 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:27:38.997087 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:27:38.997054 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:27:39.394358 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:27:39.394324 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:27:39.539165 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:27:39.539137 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:27:47.595171 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:27:47.595124 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:27:47.738499 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:27:47.738468 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:27:54.170971 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:27:54.170932 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:27:55.914828 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:27:55.914794 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:27:56.059956 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:27:56.059919 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:28:17.239244 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:28:17.239208 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:28:17.381790 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:28:17.381751 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:28:24.171607 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:28:24.171524 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:28:34.144747 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:28:34.144712 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:28:34.309873 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:28:34.309839 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:28:54.171437 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:28:54.171397 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:29:24.170876 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:29:24.170838 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:29:34.492026 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:29:34.491992 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:29:43.459579 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:29:43.459548 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 01:29:43.460131 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:29:43.459596 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 01:29:45.488653 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:29:45.488609 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:29:54.170745 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:29:54.170637 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:30:10.846147 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:10.846109 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:11.011382 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:11.011336 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:11.208782 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:11.208658 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:12.455224 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:12.455181 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:12.605034 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:12.605003 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:13.706978 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:13.706942 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:19.428129 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:19.428094 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:19.580420 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:19.580389 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:20.770074 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:20.770030 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:20.983399 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:20.983367 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2"] Nov 02 01:30:20.983755 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:20.983741 5044 cpu_manager.go:414] "containerMap: removing container" podUID="d5c07387-ae45-48d3-8b9d-8f328bdfd63c" containerName="registry" Nov 02 01:30:20.983813 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:20.983756 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c07387-ae45-48d3-8b9d-8f328bdfd63c" containerName="registry" Nov 02 01:30:20.983813 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:20.983766 5044 cpu_manager.go:414] "containerMap: removing container" podUID="7c2b846b-5c8b-4982-abc9-809fdbffe82d" containerName="registry" Nov 02 01:30:20.983813 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:20.983771 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c2b846b-5c8b-4982-abc9-809fdbffe82d" containerName="registry" Nov 02 01:30:20.983813 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:20.983780 5044 cpu_manager.go:414] "containerMap: removing container" podUID="eb932801-604f-430e-bfe3-8d4c9a1a7ca6" containerName="insights-gathering" Nov 02 01:30:20.983813 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:20.983785 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb932801-604f-430e-bfe3-8d4c9a1a7ca6" containerName="insights-gathering" Nov 02 01:30:20.983966 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:20.983844 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c07387-ae45-48d3-8b9d-8f328bdfd63c" containerName="registry" Nov 02 01:30:20.983966 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:20.983852 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c2b846b-5c8b-4982-abc9-809fdbffe82d" containerName="registry" Nov 02 01:30:20.983966 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:20.983859 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb932801-604f-430e-bfe3-8d4c9a1a7ca6" containerName="insights-gathering" Nov 02 01:30:20.985737 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:20.985722 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.039393 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.039312 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 01:30:21.039393 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.039340 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-t97jd\"" Nov 02 01:30:21.040254 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.040231 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:21.075138 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.075100 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d93929-7a30-4a2a-a145-9869ea9b44fe-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.075138 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.075138 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c8cz\" (UniqueName: \"kubernetes.io/projected/a9d93929-7a30-4a2a-a145-9869ea9b44fe-kube-api-access-8c8cz\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.075325 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.075267 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d93929-7a30-4a2a-a145-9869ea9b44fe-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.075325 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.075292 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d93929-7a30-4a2a-a145-9869ea9b44fe-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.075325 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.075312 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d93929-7a30-4a2a-a145-9869ea9b44fe-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.075467 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.075352 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d93929-7a30-4a2a-a145-9869ea9b44fe-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.075467 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.075442 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d93929-7a30-4a2a-a145-9869ea9b44fe-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.075529 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.075478 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d93929-7a30-4a2a-a145-9869ea9b44fe-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.075529 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.075503 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d93929-7a30-4a2a-a145-9869ea9b44fe-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.090906 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.090886 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2"] Nov 02 01:30:21.176737 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.176703 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d93929-7a30-4a2a-a145-9869ea9b44fe-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.176737 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.176737 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d93929-7a30-4a2a-a145-9869ea9b44fe-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.176960 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.176756 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d93929-7a30-4a2a-a145-9869ea9b44fe-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.176960 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.176774 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d93929-7a30-4a2a-a145-9869ea9b44fe-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.176960 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.176914 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d93929-7a30-4a2a-a145-9869ea9b44fe-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.177127 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.176965 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d93929-7a30-4a2a-a145-9869ea9b44fe-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.177127 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.177013 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d93929-7a30-4a2a-a145-9869ea9b44fe-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.177127 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.177067 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d93929-7a30-4a2a-a145-9869ea9b44fe-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.177127 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.177106 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c8cz\" (UniqueName: \"kubernetes.io/projected/a9d93929-7a30-4a2a-a145-9869ea9b44fe-kube-api-access-8c8cz\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.177329 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.177173 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d93929-7a30-4a2a-a145-9869ea9b44fe-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.177329 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.177193 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9d93929-7a30-4a2a-a145-9869ea9b44fe-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.177464 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.177447 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9d93929-7a30-4a2a-a145-9869ea9b44fe-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.177586 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.177559 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9d93929-7a30-4a2a-a145-9869ea9b44fe-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.177651 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.177608 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9d93929-7a30-4a2a-a145-9869ea9b44fe-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.179296 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.179268 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9d93929-7a30-4a2a-a145-9869ea9b44fe-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.179464 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.179446 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9d93929-7a30-4a2a-a145-9869ea9b44fe-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.190322 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.190297 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:21.239053 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.239019 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9d93929-7a30-4a2a-a145-9869ea9b44fe-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.239175 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.239156 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c8cz\" (UniqueName: \"kubernetes.io/projected/a9d93929-7a30-4a2a-a145-9869ea9b44fe-kube-api-access-8c8cz\") pod \"openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2\" (UID: \"a9d93929-7a30-4a2a-a145-9869ea9b44fe\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.297287 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.297218 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:21.719630 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.719378 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2"] Nov 02 01:30:21.721747 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:30:21.721715 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d93929_7a30_4a2a_a145_9869ea9b44fe.slice/crio-e7ffe9f7f285106f9fd462e8683164de063927af1ba4ea4d9533fa5e8581aa7e WatchSource:0}: Error finding container e7ffe9f7f285106f9fd462e8683164de063927af1ba4ea4d9533fa5e8581aa7e: Status 404 returned error can't find the container with id e7ffe9f7f285106f9fd462e8683164de063927af1ba4ea4d9533fa5e8581aa7e Nov 02 01:30:21.723658 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.723643 5044 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:30:21.916906 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:21.916867 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" event={"ID":"a9d93929-7a30-4a2a-a145-9869ea9b44fe","Type":"ContainerStarted","Data":"e7ffe9f7f285106f9fd462e8683164de063927af1ba4ea4d9533fa5e8581aa7e"} Nov 02 01:30:24.176385 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:30:24.176345 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:30:24.863766 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:24.863686 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:25.558049 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:25.558016 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:30:25.558339 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:25.558092 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:30:25.558339 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:25.558125 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:30:25.940809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:25.940710 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" event={"ID":"a9d93929-7a30-4a2a-a145-9869ea9b44fe","Type":"ContainerStarted","Data":"14a29ff642e6cc618b243686ef4f346758f1d942299cd20e0d06bddc102cc823"} Nov 02 01:30:26.165488 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:26.165431 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" podStartSLOduration=2.332162661 podStartE2EDuration="6.165417474s" totalImagesPullingTime="3.833254813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:26.11418579 +0000 UTC m=+342.785438469" watchObservedRunningTime="2026-11-02 01:30:26.165417474 +0000 UTC m=+342.836670146" Nov 02 01:30:26.297935 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:26.297897 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:26.302259 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:26.302236 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:26.944365 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:26.944334 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:26.945431 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:26.945410 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2" Nov 02 01:30:34.828375 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:34.828345 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:51.214262 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:51.214227 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:51.505133 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:51.505038 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:52.151143 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.151111 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:52.351752 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.351724 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw"] Nov 02 01:30:52.355366 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.355348 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:52.432117 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.432043 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b52f569a-5817-41f0-9cae-d51094ea35a3-webhook-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-l6tcw\" (UID: \"b52f569a-5817-41f0-9cae-d51094ea35a3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:52.432117 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.432089 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgl78\" (UniqueName: \"kubernetes.io/projected/b52f569a-5817-41f0-9cae-d51094ea35a3-kube-api-access-cgl78\") pod \"opendatahub-operator-controller-manager-5989ddd979-l6tcw\" (UID: \"b52f569a-5817-41f0-9cae-d51094ea35a3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:52.432285 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.432165 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b52f569a-5817-41f0-9cae-d51094ea35a3-manifests\") pod \"opendatahub-operator-controller-manager-5989ddd979-l6tcw\" (UID: \"b52f569a-5817-41f0-9cae-d51094ea35a3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:52.432285 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.432208 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b52f569a-5817-41f0-9cae-d51094ea35a3-metrics-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-l6tcw\" (UID: \"b52f569a-5817-41f0-9cae-d51094ea35a3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:52.432285 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.432230 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b52f569a-5817-41f0-9cae-d51094ea35a3-tmp\") pod \"opendatahub-operator-controller-manager-5989ddd979-l6tcw\" (UID: \"b52f569a-5817-41f0-9cae-d51094ea35a3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:52.432394 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.432279 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b52f569a-5817-41f0-9cae-d51094ea35a3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-l6tcw\" (UID: \"b52f569a-5817-41f0-9cae-d51094ea35a3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:52.433882 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.433864 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 01:30:52.433936 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.433865 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 01:30:52.440776 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.440758 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 01:30:52.440776 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.440759 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 01:30:52.440928 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.440915 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 01:30:52.440968 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.440923 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-vxbpb\"" Nov 02 01:30:52.443337 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.443297 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:52.533062 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.533027 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b52f569a-5817-41f0-9cae-d51094ea35a3-metrics-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-l6tcw\" (UID: \"b52f569a-5817-41f0-9cae-d51094ea35a3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:52.533062 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.533067 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b52f569a-5817-41f0-9cae-d51094ea35a3-tmp\") pod \"opendatahub-operator-controller-manager-5989ddd979-l6tcw\" (UID: \"b52f569a-5817-41f0-9cae-d51094ea35a3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:52.533279 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.533090 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b52f569a-5817-41f0-9cae-d51094ea35a3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-l6tcw\" (UID: \"b52f569a-5817-41f0-9cae-d51094ea35a3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:52.533279 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.533157 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b52f569a-5817-41f0-9cae-d51094ea35a3-webhook-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-l6tcw\" (UID: \"b52f569a-5817-41f0-9cae-d51094ea35a3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:52.533279 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.533192 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgl78\" (UniqueName: \"kubernetes.io/projected/b52f569a-5817-41f0-9cae-d51094ea35a3-kube-api-access-cgl78\") pod \"opendatahub-operator-controller-manager-5989ddd979-l6tcw\" (UID: \"b52f569a-5817-41f0-9cae-d51094ea35a3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:52.533279 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.533225 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b52f569a-5817-41f0-9cae-d51094ea35a3-manifests\") pod \"opendatahub-operator-controller-manager-5989ddd979-l6tcw\" (UID: \"b52f569a-5817-41f0-9cae-d51094ea35a3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:52.533581 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.533553 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b52f569a-5817-41f0-9cae-d51094ea35a3-manifests\") pod \"opendatahub-operator-controller-manager-5989ddd979-l6tcw\" (UID: \"b52f569a-5817-41f0-9cae-d51094ea35a3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:52.535612 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.535576 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b52f569a-5817-41f0-9cae-d51094ea35a3-tmp\") pod \"opendatahub-operator-controller-manager-5989ddd979-l6tcw\" (UID: \"b52f569a-5817-41f0-9cae-d51094ea35a3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:52.535761 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.535691 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b52f569a-5817-41f0-9cae-d51094ea35a3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-l6tcw\" (UID: \"b52f569a-5817-41f0-9cae-d51094ea35a3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:52.535761 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.535755 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b52f569a-5817-41f0-9cae-d51094ea35a3-webhook-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-l6tcw\" (UID: \"b52f569a-5817-41f0-9cae-d51094ea35a3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:52.535905 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.535881 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b52f569a-5817-41f0-9cae-d51094ea35a3-metrics-cert\") pod \"opendatahub-operator-controller-manager-5989ddd979-l6tcw\" (UID: \"b52f569a-5817-41f0-9cae-d51094ea35a3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:52.594156 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.594125 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw"] Nov 02 01:30:52.597897 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.597866 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:52.599185 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.599161 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgl78\" (UniqueName: \"kubernetes.io/projected/b52f569a-5817-41f0-9cae-d51094ea35a3-kube-api-access-cgl78\") pod \"opendatahub-operator-controller-manager-5989ddd979-l6tcw\" (UID: \"b52f569a-5817-41f0-9cae-d51094ea35a3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:52.670181 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.670151 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:52.752091 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.752067 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:52.898319 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:52.898289 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:53.211899 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:53.211867 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw"] Nov 02 01:30:53.263427 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:30:53.263399 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52f569a_5817_41f0_9cae_d51094ea35a3.slice/crio-3aa5fde1e07062e43a4025e3639d45a86abf0b23004f30d5821b45631852777e WatchSource:0}: Error finding container 3aa5fde1e07062e43a4025e3639d45a86abf0b23004f30d5821b45631852777e: Status 404 returned error can't find the container with id 3aa5fde1e07062e43a4025e3639d45a86abf0b23004f30d5821b45631852777e Nov 02 01:30:53.415238 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:53.415201 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:54.049627 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:54.049594 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" event={"ID":"b52f569a-5817-41f0-9cae-d51094ea35a3","Type":"ContainerStarted","Data":"3aa5fde1e07062e43a4025e3639d45a86abf0b23004f30d5821b45631852777e"} Nov 02 01:30:54.171582 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:30:54.171544 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:30:55.227471 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:55.227429 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:55.417720 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:55.417686 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:55.792010 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:55.791977 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:57.064929 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:57.064892 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" event={"ID":"b52f569a-5817-41f0-9cae-d51094ea35a3","Type":"ContainerStarted","Data":"970bee21fc2cc7acc05e9fbc26371299d82495a4ca57e44e0a177bb41bddd0ed"} Nov 02 01:30:58.070629 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:58.070595 5044 generic.go:356] "Generic (PLEG): container finished" podID="b52f569a-5817-41f0-9cae-d51094ea35a3" containerID="970bee21fc2cc7acc05e9fbc26371299d82495a4ca57e44e0a177bb41bddd0ed" exitCode=0 Nov 02 01:30:58.071033 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:58.070699 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" event={"ID":"b52f569a-5817-41f0-9cae-d51094ea35a3","Type":"ContainerDied","Data":"970bee21fc2cc7acc05e9fbc26371299d82495a4ca57e44e0a177bb41bddd0ed"} Nov 02 01:30:58.104192 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:58.104162 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:30:59.076788 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:59.076751 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" event={"ID":"b52f569a-5817-41f0-9cae-d51094ea35a3","Type":"ContainerStarted","Data":"603315cf20cb330128f8530584a71b8d2ea9867c5075a698ba14466cf31a4fdd"} Nov 02 01:30:59.077271 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:59.076860 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:30:59.269544 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:30:59.269496 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" podStartSLOduration=2.632264508 podStartE2EDuration="7.269482306s" totalImagesPullingTime="4.151485739s" totalInitContainerRuntime="485.732059ms" isStatefulPod=false podCreationTimestamp="2026-11-02 01:30:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:30:59.268380048 +0000 UTC m=+375.939632729" watchObservedRunningTime="2026-11-02 01:30:59.269482306 +0000 UTC m=+375.940734977" Nov 02 01:31:00.621228 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:00.621195 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:00.794414 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:00.794384 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:00.948356 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:00.948283 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:04.435018 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:04.434980 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:04.629685 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:04.629630 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:05.930568 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:05.930537 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:06.113963 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:06.113930 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:08.451258 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:08.451225 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:08.601178 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:08.601144 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:08.774841 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:08.774816 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:10.083697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:10.083644 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5989ddd979-l6tcw" Nov 02 01:31:12.990719 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:12.990685 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:14.018915 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:14.018881 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:14.173514 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:14.173488 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:14.316868 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:14.316785 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:15.329317 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:15.329288 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:15.474502 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:15.474475 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:16.051966 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:16.051937 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:16.447900 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:16.447823 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-c7fzl"] Nov 02 01:31:16.453880 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:16.453862 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-c7fzl" Nov 02 01:31:16.584931 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:16.584900 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 01:31:16.584931 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:16.584915 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 01:31:16.585158 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:16.584966 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-zdb72\"" Nov 02 01:31:16.586373 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:16.586347 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-c7fzl"] Nov 02 01:31:16.634743 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:16.634716 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jvtk\" (UniqueName: \"kubernetes.io/projected/1c258db9-4e2f-4733-be43-ccaa2bd519ab-kube-api-access-2jvtk\") pod \"cert-manager-597b95cb84-c7fzl\" (UID: \"1c258db9-4e2f-4733-be43-ccaa2bd519ab\") " pod="cert-manager/cert-manager-597b95cb84-c7fzl" Nov 02 01:31:16.634878 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:16.634860 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c258db9-4e2f-4733-be43-ccaa2bd519ab-bound-sa-token\") pod \"cert-manager-597b95cb84-c7fzl\" (UID: \"1c258db9-4e2f-4733-be43-ccaa2bd519ab\") " pod="cert-manager/cert-manager-597b95cb84-c7fzl" Nov 02 01:31:16.736048 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:16.735978 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jvtk\" (UniqueName: \"kubernetes.io/projected/1c258db9-4e2f-4733-be43-ccaa2bd519ab-kube-api-access-2jvtk\") pod \"cert-manager-597b95cb84-c7fzl\" (UID: \"1c258db9-4e2f-4733-be43-ccaa2bd519ab\") " pod="cert-manager/cert-manager-597b95cb84-c7fzl" Nov 02 01:31:16.736165 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:16.736049 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c258db9-4e2f-4733-be43-ccaa2bd519ab-bound-sa-token\") pod \"cert-manager-597b95cb84-c7fzl\" (UID: \"1c258db9-4e2f-4733-be43-ccaa2bd519ab\") " pod="cert-manager/cert-manager-597b95cb84-c7fzl" Nov 02 01:31:16.798101 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:16.798072 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c258db9-4e2f-4733-be43-ccaa2bd519ab-bound-sa-token\") pod \"cert-manager-597b95cb84-c7fzl\" (UID: \"1c258db9-4e2f-4733-be43-ccaa2bd519ab\") " pod="cert-manager/cert-manager-597b95cb84-c7fzl" Nov 02 01:31:16.798225 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:16.798133 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jvtk\" (UniqueName: \"kubernetes.io/projected/1c258db9-4e2f-4733-be43-ccaa2bd519ab-kube-api-access-2jvtk\") pod \"cert-manager-597b95cb84-c7fzl\" (UID: \"1c258db9-4e2f-4733-be43-ccaa2bd519ab\") " pod="cert-manager/cert-manager-597b95cb84-c7fzl" Nov 02 01:31:17.076787 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:17.076757 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-c7fzl" Nov 02 01:31:17.451278 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:31:17.451247 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c258db9_4e2f_4733_be43_ccaa2bd519ab.slice/crio-3d121c699cef9701f20662400775578049f568a149300f67d022c669b5bf88d1 WatchSource:0}: Error finding container 3d121c699cef9701f20662400775578049f568a149300f67d022c669b5bf88d1: Status 404 returned error can't find the container with id 3d121c699cef9701f20662400775578049f568a149300f67d022c669b5bf88d1 Nov 02 01:31:17.453428 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:17.453405 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-c7fzl"] Nov 02 01:31:17.863566 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:17.863534 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-dzndg"] Nov 02 01:31:17.868040 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:17.868022 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dzndg" Nov 02 01:31:17.922231 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:17.922205 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 01:31:17.972858 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:17.972828 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-99f97\"" Nov 02 01:31:17.972858 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:17.972848 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 01:31:17.974354 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:17.974335 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-dzndg"] Nov 02 01:31:18.047171 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:18.047141 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzfb2\" (UniqueName: \"kubernetes.io/projected/686cd2a7-b05e-45c4-af71-d0d7a6997a8c-kube-api-access-fzfb2\") pod \"authorino-operator-7b7bb4bc77-dzndg\" (UID: \"686cd2a7-b05e-45c4-af71-d0d7a6997a8c\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-dzndg" Nov 02 01:31:18.148058 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:18.147992 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzfb2\" (UniqueName: \"kubernetes.io/projected/686cd2a7-b05e-45c4-af71-d0d7a6997a8c-kube-api-access-fzfb2\") pod \"authorino-operator-7b7bb4bc77-dzndg\" (UID: \"686cd2a7-b05e-45c4-af71-d0d7a6997a8c\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-dzndg" Nov 02 01:31:18.156728 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:18.156702 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-c7fzl" event={"ID":"1c258db9-4e2f-4733-be43-ccaa2bd519ab","Type":"ContainerStarted","Data":"3d121c699cef9701f20662400775578049f568a149300f67d022c669b5bf88d1"} Nov 02 01:31:18.221922 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:18.221896 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzfb2\" (UniqueName: \"kubernetes.io/projected/686cd2a7-b05e-45c4-af71-d0d7a6997a8c-kube-api-access-fzfb2\") pod \"authorino-operator-7b7bb4bc77-dzndg\" (UID: \"686cd2a7-b05e-45c4-af71-d0d7a6997a8c\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-dzndg" Nov 02 01:31:18.481361 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:18.481280 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dzndg" Nov 02 01:31:18.856474 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:18.856393 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-dzndg"] Nov 02 01:31:18.858886 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:31:18.858860 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod686cd2a7_b05e_45c4_af71_d0d7a6997a8c.slice/crio-58dee93157c28460604f4dedf713b7639f2d90572e4b7113e3c44b8443864627 WatchSource:0}: Error finding container 58dee93157c28460604f4dedf713b7639f2d90572e4b7113e3c44b8443864627: Status 404 returned error can't find the container with id 58dee93157c28460604f4dedf713b7639f2d90572e4b7113e3c44b8443864627 Nov 02 01:31:19.163400 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:19.163315 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dzndg" event={"ID":"686cd2a7-b05e-45c4-af71-d0d7a6997a8c","Type":"ContainerStarted","Data":"58dee93157c28460604f4dedf713b7639f2d90572e4b7113e3c44b8443864627"} Nov 02 01:31:19.204640 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:19.204606 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:21.685453 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:21.684081 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-7c9h7"] Nov 02 01:31:21.687591 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:21.687569 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7c9h7" Nov 02 01:31:21.774795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:21.774762 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcqrr\" (UniqueName: \"kubernetes.io/projected/bafa8ecd-2b0c-4f80-b065-edabb802652f-kube-api-access-gcqrr\") pod \"limitador-operator-controller-manager-5778bb4449-7c9h7\" (UID: \"bafa8ecd-2b0c-4f80-b065-edabb802652f\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7c9h7" Nov 02 01:31:21.794838 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:21.794787 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-wbbr5\"" Nov 02 01:31:21.796317 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:21.796289 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-7c9h7"] Nov 02 01:31:21.875461 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:21.875434 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcqrr\" (UniqueName: \"kubernetes.io/projected/bafa8ecd-2b0c-4f80-b065-edabb802652f-kube-api-access-gcqrr\") pod \"limitador-operator-controller-manager-5778bb4449-7c9h7\" (UID: \"bafa8ecd-2b0c-4f80-b065-edabb802652f\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7c9h7" Nov 02 01:31:21.939057 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:21.938979 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcqrr\" (UniqueName: \"kubernetes.io/projected/bafa8ecd-2b0c-4f80-b065-edabb802652f-kube-api-access-gcqrr\") pod \"limitador-operator-controller-manager-5778bb4449-7c9h7\" (UID: \"bafa8ecd-2b0c-4f80-b065-edabb802652f\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7c9h7" Nov 02 01:31:22.003796 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:22.003764 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7c9h7" Nov 02 01:31:22.181609 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:22.181570 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-c7fzl" event={"ID":"1c258db9-4e2f-4733-be43-ccaa2bd519ab","Type":"ContainerStarted","Data":"18d8d96ce6fce9d80c50354823f899f763e2058bf9f0980d874c6ccec223c691"} Nov 02 01:31:22.305995 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:22.305932 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-c7fzl" podStartSLOduration=1.953330183 podStartE2EDuration="6.305913747s" totalImagesPullingTime="4.352583564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:22.305058436 +0000 UTC m=+398.976311119" watchObservedRunningTime="2026-11-02 01:31:22.305913747 +0000 UTC m=+398.977166430" Nov 02 01:31:22.468158 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:22.468135 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-7c9h7"] Nov 02 01:31:22.470469 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:31:22.470442 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbafa8ecd_2b0c_4f80_b065_edabb802652f.slice/crio-b7f4f9333fa1228256d30286183ed7c451e9554921d01a1095ad1a0a03bda6a9 WatchSource:0}: Error finding container b7f4f9333fa1228256d30286183ed7c451e9554921d01a1095ad1a0a03bda6a9: Status 404 returned error can't find the container with id b7f4f9333fa1228256d30286183ed7c451e9554921d01a1095ad1a0a03bda6a9 Nov 02 01:31:22.801536 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:22.801511 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:23.186895 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:23.186860 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7c9h7" event={"ID":"bafa8ecd-2b0c-4f80-b065-edabb802652f","Type":"ContainerStarted","Data":"b7f4f9333fa1228256d30286183ed7c451e9554921d01a1095ad1a0a03bda6a9"} Nov 02 01:31:23.188446 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:23.188422 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dzndg" event={"ID":"686cd2a7-b05e-45c4-af71-d0d7a6997a8c","Type":"ContainerStarted","Data":"b0a575e1b37ead72dcfae2911409d200a28a0e1b62b586e455061fb87d6faf52"} Nov 02 01:31:23.328507 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:23.328453 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dzndg" podStartSLOduration=2.433609049 podStartE2EDuration="6.328437098s" totalImagesPullingTime="3.894828049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:23.326788212 +0000 UTC m=+399.998040893" watchObservedRunningTime="2026-11-02 01:31:23.328437098 +0000 UTC m=+399.999689778" Nov 02 01:31:23.653350 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:23.653317 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:23.803839 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:23.803806 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:24.171422 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:31:24.171386 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:31:24.192165 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:24.192143 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dzndg" Nov 02 01:31:25.324643 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:25.324605 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:25.478821 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:25.478780 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:25.998889 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:25.998851 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:26.202565 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:26.202476 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7c9h7" event={"ID":"bafa8ecd-2b0c-4f80-b065-edabb802652f","Type":"ContainerStarted","Data":"5860bc5bebc783063ad838985d271fc9ce6a3f7b740f0eda32f288339e0769d6"} Nov 02 01:31:26.202565 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:26.202553 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7c9h7" Nov 02 01:31:26.325064 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:26.325012 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7c9h7" podStartSLOduration=1.870114712 podStartE2EDuration="5.324999205s" totalImagesPullingTime="3.454884493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:26.324704916 +0000 UTC m=+402.995957598" watchObservedRunningTime="2026-11-02 01:31:26.324999205 +0000 UTC m=+402.996251886" Nov 02 01:31:26.393303 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:26.393274 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:26.734127 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:26.734096 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:27.441357 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:27.441294 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:27.587434 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:27.587397 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:29.561001 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:29.560972 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v"] Nov 02 01:31:29.624917 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:29.624882 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v"] Nov 02 01:31:29.625076 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:29.625008 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v" Nov 02 01:31:29.628411 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:29.628388 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b919d07-7749-4e2c-ae2b-d18da91d92ef-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mw52v\" (UID: \"1b919d07-7749-4e2c-ae2b-d18da91d92ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v" Nov 02 01:31:29.628529 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:29.628422 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmvsl\" (UniqueName: \"kubernetes.io/projected/1b919d07-7749-4e2c-ae2b-d18da91d92ef-kube-api-access-rmvsl\") pod \"kuadrant-operator-controller-manager-678d87766f-mw52v\" (UID: \"1b919d07-7749-4e2c-ae2b-d18da91d92ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v" Nov 02 01:31:29.678330 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:29.678301 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xl9wm\"" Nov 02 01:31:29.728937 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:29.728913 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b919d07-7749-4e2c-ae2b-d18da91d92ef-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mw52v\" (UID: \"1b919d07-7749-4e2c-ae2b-d18da91d92ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v" Nov 02 01:31:29.729054 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:29.728940 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmvsl\" (UniqueName: \"kubernetes.io/projected/1b919d07-7749-4e2c-ae2b-d18da91d92ef-kube-api-access-rmvsl\") pod \"kuadrant-operator-controller-manager-678d87766f-mw52v\" (UID: \"1b919d07-7749-4e2c-ae2b-d18da91d92ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v" Nov 02 01:31:29.729284 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:29.729266 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b919d07-7749-4e2c-ae2b-d18da91d92ef-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-mw52v\" (UID: \"1b919d07-7749-4e2c-ae2b-d18da91d92ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v" Nov 02 01:31:29.791221 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:29.791197 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmvsl\" (UniqueName: \"kubernetes.io/projected/1b919d07-7749-4e2c-ae2b-d18da91d92ef-kube-api-access-rmvsl\") pod \"kuadrant-operator-controller-manager-678d87766f-mw52v\" (UID: \"1b919d07-7749-4e2c-ae2b-d18da91d92ef\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v" Nov 02 01:31:29.957108 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:29.957019 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v" Nov 02 01:31:30.428034 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:30.428012 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v"] Nov 02 01:31:30.430227 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:31:30.430198 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b919d07_7749_4e2c_ae2b_d18da91d92ef.slice/crio-8e8e087cffd55df18f776039785e4fa185b437cd37504efcc916a25307bc5dde WatchSource:0}: Error finding container 8e8e087cffd55df18f776039785e4fa185b437cd37504efcc916a25307bc5dde: Status 404 returned error can't find the container with id 8e8e087cffd55df18f776039785e4fa185b437cd37504efcc916a25307bc5dde Nov 02 01:31:31.226087 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:31.226053 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v" event={"ID":"1b919d07-7749-4e2c-ae2b-d18da91d92ef","Type":"ContainerStarted","Data":"8e8e087cffd55df18f776039785e4fa185b437cd37504efcc916a25307bc5dde"} Nov 02 01:31:31.398968 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:31.398934 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:31.541203 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:31.541170 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:32.012205 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:32.012175 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:32.167985 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:32.167930 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:34.408042 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:34.407997 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:34.547763 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:34.547727 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:35.166837 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:35.166805 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:35.198557 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:35.198532 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-dzndg" Nov 02 01:31:35.355398 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:35.355364 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:36.669889 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:36.669861 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:36.816300 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:36.816272 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:37.179445 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:37.179402 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:37.210330 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:37.210298 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7c9h7" Nov 02 01:31:37.264587 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:37.264550 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v" event={"ID":"1b919d07-7749-4e2c-ae2b-d18da91d92ef","Type":"ContainerStarted","Data":"dc6a1d6da10e869775ee6b94b26dda96768dbca2cc2dee87612be1ff98fbcbb5"} Nov 02 01:31:37.264771 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:37.264664 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v" Nov 02 01:31:37.485923 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:37.485825 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v" podStartSLOduration=2.215508575 podStartE2EDuration="8.485811517s" totalImagesPullingTime="6.270302942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:37.484389947 +0000 UTC m=+414.155642631" watchObservedRunningTime="2026-11-02 01:31:37.485811517 +0000 UTC m=+414.157064197" Nov 02 01:31:37.497069 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:37.497042 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:37.641369 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:37.641333 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:38.803953 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:38.803919 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:39.008446 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:39.008414 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:39.159762 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:39.159658 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:39.486332 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:39.486235 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:39.697658 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:39.697617 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:42.085285 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:42.085251 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:42.347603 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:42.347522 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:42.512897 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:42.512864 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:43.425134 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:43.425101 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:43.461819 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:43.461793 5044 scope.go:122] "RemoveContainer" containerID="ac97acf85f068fecb8405fa16c4e5ad98cf2b3ba278f95bb9c4640c3ff532ab9" Nov 02 01:31:43.567229 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:43.567185 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:43.759032 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:43.759011 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:43.903018 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:43.902987 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:44.474495 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:44.474466 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:44.627860 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:44.627832 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:44.774120 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:44.774088 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:46.035444 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.035412 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf"] Nov 02 01:31:46.039126 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.039107 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf" Nov 02 01:31:46.047901 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.047878 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3c0015e6-583e-4e3f-87ea-9f4c7229c4d8-manager-config\") pod \"jobset-controller-manager-7c89c7db4-4thbf\" (UID: \"3c0015e6-583e-4e3f-87ea-9f4c7229c4d8\") " pod="openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf" Nov 02 01:31:46.048002 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.047921 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6zvw\" (UniqueName: \"kubernetes.io/projected/3c0015e6-583e-4e3f-87ea-9f4c7229c4d8-kube-api-access-c6zvw\") pod \"jobset-controller-manager-7c89c7db4-4thbf\" (UID: \"3c0015e6-583e-4e3f-87ea-9f4c7229c4d8\") " pod="openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf" Nov 02 01:31:46.048045 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.048015 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c0015e6-583e-4e3f-87ea-9f4c7229c4d8-metrics-certs\") pod \"jobset-controller-manager-7c89c7db4-4thbf\" (UID: \"3c0015e6-583e-4e3f-87ea-9f4c7229c4d8\") " pod="openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf" Nov 02 01:31:46.048086 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.048061 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c0015e6-583e-4e3f-87ea-9f4c7229c4d8-cert\") pod \"jobset-controller-manager-7c89c7db4-4thbf\" (UID: \"3c0015e6-583e-4e3f-87ea-9f4c7229c4d8\") " pod="openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf" Nov 02 01:31:46.146125 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.146090 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 02 01:31:46.146276 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.146125 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 01:31:46.146276 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.146237 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 02 01:31:46.146276 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.146243 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-qdgzj\"" Nov 02 01:31:46.146276 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.146269 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 01:31:46.146431 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.146289 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 02 01:31:46.147636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.147618 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf"] Nov 02 01:31:46.148678 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.148656 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c0015e6-583e-4e3f-87ea-9f4c7229c4d8-metrics-certs\") pod \"jobset-controller-manager-7c89c7db4-4thbf\" (UID: \"3c0015e6-583e-4e3f-87ea-9f4c7229c4d8\") " pod="openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf" Nov 02 01:31:46.148723 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.148705 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c0015e6-583e-4e3f-87ea-9f4c7229c4d8-cert\") pod \"jobset-controller-manager-7c89c7db4-4thbf\" (UID: \"3c0015e6-583e-4e3f-87ea-9f4c7229c4d8\") " pod="openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf" Nov 02 01:31:46.148764 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.148751 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3c0015e6-583e-4e3f-87ea-9f4c7229c4d8-manager-config\") pod \"jobset-controller-manager-7c89c7db4-4thbf\" (UID: \"3c0015e6-583e-4e3f-87ea-9f4c7229c4d8\") " pod="openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf" Nov 02 01:31:46.148806 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.148774 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6zvw\" (UniqueName: \"kubernetes.io/projected/3c0015e6-583e-4e3f-87ea-9f4c7229c4d8-kube-api-access-c6zvw\") pod \"jobset-controller-manager-7c89c7db4-4thbf\" (UID: \"3c0015e6-583e-4e3f-87ea-9f4c7229c4d8\") " pod="openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf" Nov 02 01:31:46.149505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.149486 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3c0015e6-583e-4e3f-87ea-9f4c7229c4d8-manager-config\") pod \"jobset-controller-manager-7c89c7db4-4thbf\" (UID: \"3c0015e6-583e-4e3f-87ea-9f4c7229c4d8\") " pod="openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf" Nov 02 01:31:46.151649 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.151628 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c0015e6-583e-4e3f-87ea-9f4c7229c4d8-cert\") pod \"jobset-controller-manager-7c89c7db4-4thbf\" (UID: \"3c0015e6-583e-4e3f-87ea-9f4c7229c4d8\") " pod="openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf" Nov 02 01:31:46.151745 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.151717 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c0015e6-583e-4e3f-87ea-9f4c7229c4d8-metrics-certs\") pod \"jobset-controller-manager-7c89c7db4-4thbf\" (UID: \"3c0015e6-583e-4e3f-87ea-9f4c7229c4d8\") " pod="openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf" Nov 02 01:31:46.210625 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.210596 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6zvw\" (UniqueName: \"kubernetes.io/projected/3c0015e6-583e-4e3f-87ea-9f4c7229c4d8-kube-api-access-c6zvw\") pod \"jobset-controller-manager-7c89c7db4-4thbf\" (UID: \"3c0015e6-583e-4e3f-87ea-9f4c7229c4d8\") " pod="openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf" Nov 02 01:31:46.350022 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.349948 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf" Nov 02 01:31:46.585029 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.585000 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:46.720338 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.720272 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf"] Nov 02 01:31:46.723222 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:31:46.723184 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0015e6_583e_4e3f_87ea_9f4c7229c4d8.slice/crio-cb9d8dfaa5051c1717a174108b97cff3be60f909150a37670bb9b0d220a9c677 WatchSource:0}: Error finding container cb9d8dfaa5051c1717a174108b97cff3be60f909150a37670bb9b0d220a9c677: Status 404 returned error can't find the container with id cb9d8dfaa5051c1717a174108b97cff3be60f909150a37670bb9b0d220a9c677 Nov 02 01:31:46.729842 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.729819 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:46.875798 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:46.875769 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:47.310491 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:47.310452 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf" event={"ID":"3c0015e6-583e-4e3f-87ea-9f4c7229c4d8","Type":"ContainerStarted","Data":"cb9d8dfaa5051c1717a174108b97cff3be60f909150a37670bb9b0d220a9c677"} Nov 02 01:31:48.273039 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:48.273008 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v" Nov 02 01:31:50.422731 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:50.422698 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:50.566123 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:50.566083 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:51.331474 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:51.331442 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf" event={"ID":"3c0015e6-583e-4e3f-87ea-9f4c7229c4d8","Type":"ContainerStarted","Data":"adccc10b6e4cd89c8798e063f66ac8406b94f6bf4cfb2e3165581b9b6ee4e9e7"} Nov 02 01:31:51.331640 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:51.331593 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf" Nov 02 01:31:51.450605 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:51.450548 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf" podStartSLOduration=2.333009261 podStartE2EDuration="6.4505312s" totalImagesPullingTime="4.117521939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:31:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:31:51.449463771 +0000 UTC m=+428.120716446" watchObservedRunningTime="2026-11-02 01:31:51.4505312 +0000 UTC m=+428.121783880" Nov 02 01:31:54.053000 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:54.052961 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:54.171867 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:31:54.171834 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:31:54.195951 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:54.195921 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:57.057025 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:57.056974 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:57.197417 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:57.197379 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:58.035970 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:58.035935 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:31:58.193416 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:31:58.193384 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:32:02.342011 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:32:02.341978 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-7c89c7db4-4thbf" Nov 02 01:32:02.673310 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:32:02.673229 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:32:02.819494 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:32:02.819444 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:32:04.216915 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:32:04.216881 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:32:04.366819 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:32:04.366787 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:32:04.961158 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:32:04.961116 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:32:24.171563 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:32:24.171530 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:32:25.244949 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:32:25.244915 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:32:25.391528 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:32:25.391499 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:32:25.541116 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:32:25.541063 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:32:25.687931 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:32:25.687904 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:32:29.455666 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:32:29.455635 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:32:29.628192 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:32:29.628163 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:32:30.202998 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:32:30.202968 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:32:30.350907 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:32:30.350871 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:32:32.596344 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:32:32.596314 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:32:32.799030 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:32:32.799000 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:32:35.510576 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:32:35.510541 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:32:54.171562 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:32:54.171480 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:32:59.141734 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:32:59.141699 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:32:59.372895 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:32:59.372854 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:01.413202 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:01.413163 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:01.575994 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:01.575961 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:01.722126 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:01.722038 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:01.951174 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:01.951142 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:11.940397 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:11.940353 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:12.204124 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:12.204032 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:12.353889 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:12.353861 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:12.506699 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:12.506608 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:12.659224 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:12.659197 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:12.806659 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:12.806630 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:12.960179 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:12.960148 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:13.125393 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.125317 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:13.278974 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.278943 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:13.332257 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.332226 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xvcfj"] Nov 02 01:33:13.336040 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.336020 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-xvcfj" Nov 02 01:33:13.392329 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.392272 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 01:33:13.443099 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.443068 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 01:33:13.443270 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.443254 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-sl48c\"" Nov 02 01:33:13.443337 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.443325 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 01:33:13.450592 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.450571 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xvcfj"] Nov 02 01:33:13.450710 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.450614 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-s7zvr"] Nov 02 01:33:13.453856 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.453842 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-s7zvr" Nov 02 01:33:13.508230 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.508196 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f14b189b-bc74-4186-b2b5-c20db167e680-tls-certs\") pod \"model-serving-api-75bd89fdfc-xvcfj\" (UID: \"f14b189b-bc74-4186-b2b5-c20db167e680\") " pod="opendatahub/model-serving-api-75bd89fdfc-xvcfj" Nov 02 01:33:13.508395 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.508363 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vkr9\" (UniqueName: \"kubernetes.io/projected/f14b189b-bc74-4186-b2b5-c20db167e680-kube-api-access-4vkr9\") pod \"model-serving-api-75bd89fdfc-xvcfj\" (UID: \"f14b189b-bc74-4186-b2b5-c20db167e680\") " pod="opendatahub/model-serving-api-75bd89fdfc-xvcfj" Nov 02 01:33:13.508732 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.508710 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 02 01:33:13.508908 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.508893 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-tgwh2\"" Nov 02 01:33:13.510483 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.510462 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-s7zvr"] Nov 02 01:33:13.547075 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.547052 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 02 01:33:13.609620 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.609593 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq7m9\" (UniqueName: \"kubernetes.io/projected/9a000a5f-a242-4f5c-a7ee-335e84671c33-kube-api-access-sq7m9\") pod \"odh-model-controller-769875db56-s7zvr\" (UID: \"9a000a5f-a242-4f5c-a7ee-335e84671c33\") " pod="opendatahub/odh-model-controller-769875db56-s7zvr" Nov 02 01:33:13.609768 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.609648 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vkr9\" (UniqueName: \"kubernetes.io/projected/f14b189b-bc74-4186-b2b5-c20db167e680-kube-api-access-4vkr9\") pod \"model-serving-api-75bd89fdfc-xvcfj\" (UID: \"f14b189b-bc74-4186-b2b5-c20db167e680\") " pod="opendatahub/model-serving-api-75bd89fdfc-xvcfj" Nov 02 01:33:13.609768 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.609702 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a000a5f-a242-4f5c-a7ee-335e84671c33-metrics-certs\") pod \"odh-model-controller-769875db56-s7zvr\" (UID: \"9a000a5f-a242-4f5c-a7ee-335e84671c33\") " pod="opendatahub/odh-model-controller-769875db56-s7zvr" Nov 02 01:33:13.609916 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.609897 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a000a5f-a242-4f5c-a7ee-335e84671c33-cert\") pod \"odh-model-controller-769875db56-s7zvr\" (UID: \"9a000a5f-a242-4f5c-a7ee-335e84671c33\") " pod="opendatahub/odh-model-controller-769875db56-s7zvr" Nov 02 01:33:13.609974 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.609943 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f14b189b-bc74-4186-b2b5-c20db167e680-tls-certs\") pod \"model-serving-api-75bd89fdfc-xvcfj\" (UID: \"f14b189b-bc74-4186-b2b5-c20db167e680\") " pod="opendatahub/model-serving-api-75bd89fdfc-xvcfj" Nov 02 01:33:13.612381 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.612360 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f14b189b-bc74-4186-b2b5-c20db167e680-tls-certs\") pod \"model-serving-api-75bd89fdfc-xvcfj\" (UID: \"f14b189b-bc74-4186-b2b5-c20db167e680\") " pod="opendatahub/model-serving-api-75bd89fdfc-xvcfj" Nov 02 01:33:13.672238 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.672171 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vkr9\" (UniqueName: \"kubernetes.io/projected/f14b189b-bc74-4186-b2b5-c20db167e680-kube-api-access-4vkr9\") pod \"model-serving-api-75bd89fdfc-xvcfj\" (UID: \"f14b189b-bc74-4186-b2b5-c20db167e680\") " pod="opendatahub/model-serving-api-75bd89fdfc-xvcfj" Nov 02 01:33:13.711093 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.711067 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq7m9\" (UniqueName: \"kubernetes.io/projected/9a000a5f-a242-4f5c-a7ee-335e84671c33-kube-api-access-sq7m9\") pod \"odh-model-controller-769875db56-s7zvr\" (UID: \"9a000a5f-a242-4f5c-a7ee-335e84671c33\") " pod="opendatahub/odh-model-controller-769875db56-s7zvr" Nov 02 01:33:13.711219 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.711107 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a000a5f-a242-4f5c-a7ee-335e84671c33-metrics-certs\") pod \"odh-model-controller-769875db56-s7zvr\" (UID: \"9a000a5f-a242-4f5c-a7ee-335e84671c33\") " pod="opendatahub/odh-model-controller-769875db56-s7zvr" Nov 02 01:33:13.711219 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.711147 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a000a5f-a242-4f5c-a7ee-335e84671c33-cert\") pod \"odh-model-controller-769875db56-s7zvr\" (UID: \"9a000a5f-a242-4f5c-a7ee-335e84671c33\") " pod="opendatahub/odh-model-controller-769875db56-s7zvr" Nov 02 01:33:13.713586 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.713559 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a000a5f-a242-4f5c-a7ee-335e84671c33-cert\") pod \"odh-model-controller-769875db56-s7zvr\" (UID: \"9a000a5f-a242-4f5c-a7ee-335e84671c33\") " pod="opendatahub/odh-model-controller-769875db56-s7zvr" Nov 02 01:33:13.713725 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.713617 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a000a5f-a242-4f5c-a7ee-335e84671c33-metrics-certs\") pod \"odh-model-controller-769875db56-s7zvr\" (UID: \"9a000a5f-a242-4f5c-a7ee-335e84671c33\") " pod="opendatahub/odh-model-controller-769875db56-s7zvr" Nov 02 01:33:13.773432 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.773400 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq7m9\" (UniqueName: \"kubernetes.io/projected/9a000a5f-a242-4f5c-a7ee-335e84671c33-kube-api-access-sq7m9\") pod \"odh-model-controller-769875db56-s7zvr\" (UID: \"9a000a5f-a242-4f5c-a7ee-335e84671c33\") " pod="opendatahub/odh-model-controller-769875db56-s7zvr" Nov 02 01:33:13.949168 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:13.949092 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-xvcfj" Nov 02 01:33:14.066249 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:14.066217 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-s7zvr" Nov 02 01:33:14.319950 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:14.319925 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-xvcfj"] Nov 02 01:33:14.322810 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:33:14.322779 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14b189b_bc74_4186_b2b5_c20db167e680.slice/crio-b3ccc6b2c99c70b474ac9ba572c09a830d8e85052562fccfb7963a949c29d623 WatchSource:0}: Error finding container b3ccc6b2c99c70b474ac9ba572c09a830d8e85052562fccfb7963a949c29d623: Status 404 returned error can't find the container with id b3ccc6b2c99c70b474ac9ba572c09a830d8e85052562fccfb7963a949c29d623 Nov 02 01:33:14.437529 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:14.437508 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-s7zvr"] Nov 02 01:33:14.439427 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:33:14.439400 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a000a5f_a242_4f5c_a7ee_335e84671c33.slice/crio-3d7d50f94b47a17d33d8c4497b8d719dbb85bad341c0fdde2909ea5670bb3407 WatchSource:0}: Error finding container 3d7d50f94b47a17d33d8c4497b8d719dbb85bad341c0fdde2909ea5670bb3407: Status 404 returned error can't find the container with id 3d7d50f94b47a17d33d8c4497b8d719dbb85bad341c0fdde2909ea5670bb3407 Nov 02 01:33:14.473679 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:14.473653 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:14.690259 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:14.690186 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-s7zvr" event={"ID":"9a000a5f-a242-4f5c-a7ee-335e84671c33","Type":"ContainerStarted","Data":"3d7d50f94b47a17d33d8c4497b8d719dbb85bad341c0fdde2909ea5670bb3407"} Nov 02 01:33:14.691411 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:14.691390 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-xvcfj" event={"ID":"f14b189b-bc74-4186-b2b5-c20db167e680","Type":"ContainerStarted","Data":"b3ccc6b2c99c70b474ac9ba572c09a830d8e85052562fccfb7963a949c29d623"} Nov 02 01:33:17.710893 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:17.710853 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-xvcfj" event={"ID":"f14b189b-bc74-4186-b2b5-c20db167e680","Type":"ContainerStarted","Data":"f3bc6f6ab38c62ff9e40f8cbc69d50b6a300fbbd9783e0ca7a63422b66565a24"} Nov 02 01:33:17.711321 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:17.711109 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-xvcfj" Nov 02 01:33:17.832836 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:17.832772 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-xvcfj" podStartSLOduration=1.6820745179999999 podStartE2EDuration="4.832757536s" totalImagesPullingTime="3.150683018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:17.831383521 +0000 UTC m=+514.502636193" watchObservedRunningTime="2026-11-02 01:33:17.832757536 +0000 UTC m=+514.504010216" Nov 02 01:33:18.716573 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:18.716531 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-s7zvr" event={"ID":"9a000a5f-a242-4f5c-a7ee-335e84671c33","Type":"ContainerStarted","Data":"6ad32139e384595aea80eb48a16561fd277e183d408cc6ae6a72b04b24b829ba"} Nov 02 01:33:18.716985 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:18.716810 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-s7zvr" Nov 02 01:33:18.836132 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:18.836084 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-s7zvr" podStartSLOduration=1.772626318 podStartE2EDuration="5.836070592s" totalImagesPullingTime="4.063444274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:33:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:33:18.834750868 +0000 UTC m=+515.506003555" watchObservedRunningTime="2026-11-02 01:33:18.836070592 +0000 UTC m=+515.507323273" Nov 02 01:33:19.653845 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:19.653811 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:19.799981 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:19.799949 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:19.945984 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:19.945911 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:20.106023 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:20.105987 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:20.258816 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:20.258792 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:20.406040 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:20.406005 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:20.556352 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:20.556262 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:20.705138 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:20.705108 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:20.851435 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:20.851354 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:24.175656 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:33:24.175619 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:33:28.721285 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:28.721254 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-xvcfj" Nov 02 01:33:29.723541 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:29.723513 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-s7zvr" Nov 02 01:33:35.940506 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:35.940470 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:45.051729 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:45.051698 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:33:54.171585 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:33:54.171543 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:33:55.477456 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:33:55.477426 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:34:24.171845 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:34:24.171744 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:34:28.664843 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:34:28.664806 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:34:28.813012 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:34:28.812978 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:34:36.200505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:34:36.200468 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:34:37.976917 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:34:37.976882 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:34:43.487438 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:34:43.487408 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 01:34:43.488736 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:34:43.488717 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 01:34:54.171056 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:34:54.171018 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:35:11.928345 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:11.928315 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:35:12.076938 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:12.076904 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:35:24.171391 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:35:24.171357 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:35:30.905788 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:30.905754 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:35:31.460373 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:31.460340 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:35:31.611952 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:31.611920 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:35:34.521128 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.521095 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4"] Nov 02 01:35:34.525161 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.525141 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:34.565941 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.565918 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39c708b5-209a-4b5c-9f5b-402174f822b4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4\" (UID: \"39c708b5-209a-4b5c-9f5b-402174f822b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:34.566070 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.565988 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39c708b5-209a-4b5c-9f5b-402174f822b4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4\" (UID: \"39c708b5-209a-4b5c-9f5b-402174f822b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:34.566139 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.566116 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39c708b5-209a-4b5c-9f5b-402174f822b4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4\" (UID: \"39c708b5-209a-4b5c-9f5b-402174f822b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:34.566199 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.566170 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39c708b5-209a-4b5c-9f5b-402174f822b4-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4\" (UID: \"39c708b5-209a-4b5c-9f5b-402174f822b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:34.566291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.566264 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjwt8\" (UniqueName: \"kubernetes.io/projected/39c708b5-209a-4b5c-9f5b-402174f822b4-kube-api-access-cjwt8\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4\" (UID: \"39c708b5-209a-4b5c-9f5b-402174f822b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:34.566364 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.566340 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39c708b5-209a-4b5c-9f5b-402174f822b4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4\" (UID: \"39c708b5-209a-4b5c-9f5b-402174f822b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:34.578735 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.578710 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 01:35:34.578735 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.578729 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 01:35:34.617998 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.617969 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:35:34.629367 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.629349 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kf2fq\"" Nov 02 01:35:34.629553 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.629538 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 02 01:35:34.631081 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.631064 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4"] Nov 02 01:35:34.667065 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.667040 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39c708b5-209a-4b5c-9f5b-402174f822b4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4\" (UID: \"39c708b5-209a-4b5c-9f5b-402174f822b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:34.667170 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.667069 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39c708b5-209a-4b5c-9f5b-402174f822b4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4\" (UID: \"39c708b5-209a-4b5c-9f5b-402174f822b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:34.667170 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.667092 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39c708b5-209a-4b5c-9f5b-402174f822b4-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4\" (UID: \"39c708b5-209a-4b5c-9f5b-402174f822b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:34.667170 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.667111 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjwt8\" (UniqueName: \"kubernetes.io/projected/39c708b5-209a-4b5c-9f5b-402174f822b4-kube-api-access-cjwt8\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4\" (UID: \"39c708b5-209a-4b5c-9f5b-402174f822b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:34.667170 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.667139 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39c708b5-209a-4b5c-9f5b-402174f822b4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4\" (UID: \"39c708b5-209a-4b5c-9f5b-402174f822b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:34.667348 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.667178 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39c708b5-209a-4b5c-9f5b-402174f822b4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4\" (UID: \"39c708b5-209a-4b5c-9f5b-402174f822b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:34.667571 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.667553 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39c708b5-209a-4b5c-9f5b-402174f822b4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4\" (UID: \"39c708b5-209a-4b5c-9f5b-402174f822b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:34.667650 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.667607 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39c708b5-209a-4b5c-9f5b-402174f822b4-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4\" (UID: \"39c708b5-209a-4b5c-9f5b-402174f822b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:34.667722 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.667705 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39c708b5-209a-4b5c-9f5b-402174f822b4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4\" (UID: \"39c708b5-209a-4b5c-9f5b-402174f822b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:34.669361 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.669341 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39c708b5-209a-4b5c-9f5b-402174f822b4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4\" (UID: \"39c708b5-209a-4b5c-9f5b-402174f822b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:34.669654 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.669636 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39c708b5-209a-4b5c-9f5b-402174f822b4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4\" (UID: \"39c708b5-209a-4b5c-9f5b-402174f822b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:34.729504 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.729482 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjwt8\" (UniqueName: \"kubernetes.io/projected/39c708b5-209a-4b5c-9f5b-402174f822b4-kube-api-access-cjwt8\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4\" (UID: \"39c708b5-209a-4b5c-9f5b-402174f822b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:34.836549 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:34.836486 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:35.208143 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:35.208118 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4"] Nov 02 01:35:35.209902 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:35:35.209878 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39c708b5_209a_4b5c_9f5b_402174f822b4.slice/crio-3244f6ea2a01446ee5f2da96220484d1ad68ccc2b52fc93a99dc3e5eee8aebbf WatchSource:0}: Error finding container 3244f6ea2a01446ee5f2da96220484d1ad68ccc2b52fc93a99dc3e5eee8aebbf: Status 404 returned error can't find the container with id 3244f6ea2a01446ee5f2da96220484d1ad68ccc2b52fc93a99dc3e5eee8aebbf Nov 02 01:35:35.211206 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:35.211189 5044 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:35:35.322071 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:35.322040 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" event={"ID":"39c708b5-209a-4b5c-9f5b-402174f822b4","Type":"ContainerStarted","Data":"3244f6ea2a01446ee5f2da96220484d1ad68ccc2b52fc93a99dc3e5eee8aebbf"} Nov 02 01:35:36.545793 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:36.545700 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:35:37.334313 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:37.334270 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" event={"ID":"39c708b5-209a-4b5c-9f5b-402174f822b4","Type":"ContainerStarted","Data":"45a82bfb4503e4a4ed925e97da84163505460c9c80659c7e6198e9f03b4aee62"} Nov 02 01:35:37.454853 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:37.454803 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podStartSLOduration=1.974078713 podStartE2EDuration="3.454788484s" totalImagesPullingTime="1.480709771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:37.453248969 +0000 UTC m=+654.124501651" watchObservedRunningTime="2026-11-02 01:35:37.454788484 +0000 UTC m=+654.126041162" Nov 02 01:35:37.836868 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:37.836832 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:38.924650 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:38.924607 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:35:42.991603 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:42.991567 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:35:44.836721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:44.836691 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:44.854618 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:44.854589 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:45.380028 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:45.379996 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:35:49.178418 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.178383 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj"] Nov 02 01:35:49.182004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.181988 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:49.222545 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.222518 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj"] Nov 02 01:35:49.236215 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.236195 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 02 01:35:49.264691 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.264646 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:35:49.288593 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.288570 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fkw9\" (UniqueName: \"kubernetes.io/projected/67c587ab-de84-4d10-a0de-69762a529054-kube-api-access-6fkw9\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg2fj\" (UID: \"67c587ab-de84-4d10-a0de-69762a529054\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:49.288721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.288627 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67c587ab-de84-4d10-a0de-69762a529054-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg2fj\" (UID: \"67c587ab-de84-4d10-a0de-69762a529054\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:49.288721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.288705 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67c587ab-de84-4d10-a0de-69762a529054-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg2fj\" (UID: \"67c587ab-de84-4d10-a0de-69762a529054\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:49.288800 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.288744 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67c587ab-de84-4d10-a0de-69762a529054-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg2fj\" (UID: \"67c587ab-de84-4d10-a0de-69762a529054\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:49.288800 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.288759 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67c587ab-de84-4d10-a0de-69762a529054-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg2fj\" (UID: \"67c587ab-de84-4d10-a0de-69762a529054\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:49.288861 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.288810 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67c587ab-de84-4d10-a0de-69762a529054-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg2fj\" (UID: \"67c587ab-de84-4d10-a0de-69762a529054\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:49.390218 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.390188 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67c587ab-de84-4d10-a0de-69762a529054-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg2fj\" (UID: \"67c587ab-de84-4d10-a0de-69762a529054\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:49.390218 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.390219 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67c587ab-de84-4d10-a0de-69762a529054-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg2fj\" (UID: \"67c587ab-de84-4d10-a0de-69762a529054\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:49.390449 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.390242 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67c587ab-de84-4d10-a0de-69762a529054-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg2fj\" (UID: \"67c587ab-de84-4d10-a0de-69762a529054\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:49.390449 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.390303 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fkw9\" (UniqueName: \"kubernetes.io/projected/67c587ab-de84-4d10-a0de-69762a529054-kube-api-access-6fkw9\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg2fj\" (UID: \"67c587ab-de84-4d10-a0de-69762a529054\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:49.390449 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.390363 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67c587ab-de84-4d10-a0de-69762a529054-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg2fj\" (UID: \"67c587ab-de84-4d10-a0de-69762a529054\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:49.390449 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.390387 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67c587ab-de84-4d10-a0de-69762a529054-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg2fj\" (UID: \"67c587ab-de84-4d10-a0de-69762a529054\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:49.390661 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.390645 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67c587ab-de84-4d10-a0de-69762a529054-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg2fj\" (UID: \"67c587ab-de84-4d10-a0de-69762a529054\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:49.390755 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.390653 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67c587ab-de84-4d10-a0de-69762a529054-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg2fj\" (UID: \"67c587ab-de84-4d10-a0de-69762a529054\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:49.390803 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.390787 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67c587ab-de84-4d10-a0de-69762a529054-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg2fj\" (UID: \"67c587ab-de84-4d10-a0de-69762a529054\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:49.392529 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.392508 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67c587ab-de84-4d10-a0de-69762a529054-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg2fj\" (UID: \"67c587ab-de84-4d10-a0de-69762a529054\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:49.393002 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.392982 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67c587ab-de84-4d10-a0de-69762a529054-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg2fj\" (UID: \"67c587ab-de84-4d10-a0de-69762a529054\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:49.452363 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.452307 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fkw9\" (UniqueName: \"kubernetes.io/projected/67c587ab-de84-4d10-a0de-69762a529054-kube-api-access-6fkw9\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fg2fj\" (UID: \"67c587ab-de84-4d10-a0de-69762a529054\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:49.494618 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.494589 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:49.870091 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:49.870061 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj"] Nov 02 01:35:49.871717 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:35:49.871690 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c587ab_de84_4d10_a0de_69762a529054.slice/crio-9fced0f30b74d37d9ce6a51dd0951ee960ab91d0168a60f4347b0e388fcc15d4 WatchSource:0}: Error finding container 9fced0f30b74d37d9ce6a51dd0951ee960ab91d0168a60f4347b0e388fcc15d4: Status 404 returned error can't find the container with id 9fced0f30b74d37d9ce6a51dd0951ee960ab91d0168a60f4347b0e388fcc15d4 Nov 02 01:35:50.393974 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:50.393934 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" event={"ID":"67c587ab-de84-4d10-a0de-69762a529054","Type":"ContainerStarted","Data":"9fced0f30b74d37d9ce6a51dd0951ee960ab91d0168a60f4347b0e388fcc15d4"} Nov 02 01:35:51.399991 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:51.399957 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" event={"ID":"67c587ab-de84-4d10-a0de-69762a529054","Type":"ContainerStarted","Data":"163c19da2966e4cd84454b2a4c3456e589b27e75f83c5632e8876561c3934a4c"} Nov 02 01:35:51.494978 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:51.494952 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:51.521562 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:51.521516 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" podStartSLOduration=1.9285254219999999 podStartE2EDuration="2.521503483s" totalImagesPullingTime="592.978061ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:51.52001911 +0000 UTC m=+668.191271792" watchObservedRunningTime="2026-11-02 01:35:51.521503483 +0000 UTC m=+668.192756164" Nov 02 01:35:52.304349 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:52.304321 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:35:53.443617 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.443585 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54"] Nov 02 01:35:53.447173 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.447155 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.501029 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.501003 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pjf7g\"" Nov 02 01:35:53.526391 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.526359 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.526535 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.526397 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.526535 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.526427 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.526535 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.526480 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832cc120-3c4e-4605-b990-683ca969b6ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.526535 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.526497 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.526809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.526594 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832cc120-3c4e-4605-b990-683ca969b6ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.526809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.526733 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb5cs\" (UniqueName: \"kubernetes.io/projected/832cc120-3c4e-4605-b990-683ca969b6ab-kube-api-access-kb5cs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.526809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.526769 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832cc120-3c4e-4605-b990-683ca969b6ab-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.526809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.526804 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.554028 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.553996 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54"] Nov 02 01:35:53.627532 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.627498 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832cc120-3c4e-4605-b990-683ca969b6ab-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.627738 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.627549 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.627738 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.627616 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.627738 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.627651 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.627738 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.627703 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.627965 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.627736 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832cc120-3c4e-4605-b990-683ca969b6ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.627965 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.627761 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.627965 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.627784 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832cc120-3c4e-4605-b990-683ca969b6ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.627965 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.627842 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb5cs\" (UniqueName: \"kubernetes.io/projected/832cc120-3c4e-4605-b990-683ca969b6ab-kube-api-access-kb5cs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.628390 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.628028 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.628390 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.628128 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.628390 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.628255 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.628547 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.628526 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.628694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.628655 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832cc120-3c4e-4605-b990-683ca969b6ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.630015 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.629993 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.630325 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.630307 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832cc120-3c4e-4605-b990-683ca969b6ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.689819 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.689794 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832cc120-3c4e-4605-b990-683ca969b6ab-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.689945 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.689926 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb5cs\" (UniqueName: \"kubernetes.io/projected/832cc120-3c4e-4605-b990-683ca969b6ab-kube-api-access-kb5cs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-w4b54\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:53.759701 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:53.759682 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:54.171615 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:35:54.171542 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:35:54.186653 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:54.186629 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54"] Nov 02 01:35:54.188218 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:35:54.188194 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod832cc120_3c4e_4605_b990_683ca969b6ab.slice/crio-caca70f04d3fd22470e2b6ca52602e75bf6928f3433f710da6336dc3cb38ca77 WatchSource:0}: Error finding container caca70f04d3fd22470e2b6ca52602e75bf6928f3433f710da6336dc3cb38ca77: Status 404 returned error can't find the container with id caca70f04d3fd22470e2b6ca52602e75bf6928f3433f710da6336dc3cb38ca77 Nov 02 01:35:54.190352 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:54.190324 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:35:54.190423 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:54.190387 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:35:54.190423 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:54.190415 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:35:54.415469 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:54.415429 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" event={"ID":"832cc120-3c4e-4605-b990-683ca969b6ab","Type":"ContainerStarted","Data":"36d28374340aeb8a48afc28c28a583d1db1ec6fbcc50db6cb1ad908ea09631b2"} Nov 02 01:35:54.415627 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:54.415522 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" event={"ID":"832cc120-3c4e-4605-b990-683ca969b6ab","Type":"ContainerStarted","Data":"caca70f04d3fd22470e2b6ca52602e75bf6928f3433f710da6336dc3cb38ca77"} Nov 02 01:35:54.539186 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:54.539134 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" podStartSLOduration=1.539120024 podStartE2EDuration="1.539120024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:35:54.537740286 +0000 UTC m=+671.208992968" watchObservedRunningTime="2026-11-02 01:35:54.539120024 +0000 UTC m=+671.210372704" Nov 02 01:35:54.759840 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:54.759808 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:35:55.764809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:55.764772 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-w4b54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 02 01:35:55.765414 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:55.765365 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" podUID="832cc120-3c4e-4605-b990-683ca969b6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 02 01:35:56.739734 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.739692 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-p9dmt"] Nov 02 01:35:56.743152 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.743135 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:35:56.760568 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.760547 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-w4b54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 02 01:35:56.760651 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.760585 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" podUID="832cc120-3c4e-4605-b990-683ca969b6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 02 01:35:56.834244 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.834202 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:35:56.846898 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.846874 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 02 01:35:56.848912 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.848893 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-p9dmt"] Nov 02 01:35:56.859600 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.859566 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0981d7f6-9826-4e4d-a643-09a2514967d0-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-p9dmt\" (UID: \"0981d7f6-9826-4e4d-a643-09a2514967d0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:35:56.859600 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.859591 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0981d7f6-9826-4e4d-a643-09a2514967d0-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-p9dmt\" (UID: \"0981d7f6-9826-4e4d-a643-09a2514967d0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:35:56.859784 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.859637 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8csx\" (UniqueName: \"kubernetes.io/projected/0981d7f6-9826-4e4d-a643-09a2514967d0-kube-api-access-n8csx\") pod \"echo-simulator-kserve-64b6f8f96c-p9dmt\" (UID: \"0981d7f6-9826-4e4d-a643-09a2514967d0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:35:56.859784 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.859768 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0981d7f6-9826-4e4d-a643-09a2514967d0-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-p9dmt\" (UID: \"0981d7f6-9826-4e4d-a643-09a2514967d0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:35:56.859906 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.859799 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0981d7f6-9826-4e4d-a643-09a2514967d0-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-p9dmt\" (UID: \"0981d7f6-9826-4e4d-a643-09a2514967d0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:35:56.859906 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.859856 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0981d7f6-9826-4e4d-a643-09a2514967d0-home\") pod \"echo-simulator-kserve-64b6f8f96c-p9dmt\" (UID: \"0981d7f6-9826-4e4d-a643-09a2514967d0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:35:56.969693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.961040 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8csx\" (UniqueName: \"kubernetes.io/projected/0981d7f6-9826-4e4d-a643-09a2514967d0-kube-api-access-n8csx\") pod \"echo-simulator-kserve-64b6f8f96c-p9dmt\" (UID: \"0981d7f6-9826-4e4d-a643-09a2514967d0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:35:56.969693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.961138 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0981d7f6-9826-4e4d-a643-09a2514967d0-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-p9dmt\" (UID: \"0981d7f6-9826-4e4d-a643-09a2514967d0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:35:56.969693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.961172 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0981d7f6-9826-4e4d-a643-09a2514967d0-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-p9dmt\" (UID: \"0981d7f6-9826-4e4d-a643-09a2514967d0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:35:56.969693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.961238 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0981d7f6-9826-4e4d-a643-09a2514967d0-home\") pod \"echo-simulator-kserve-64b6f8f96c-p9dmt\" (UID: \"0981d7f6-9826-4e4d-a643-09a2514967d0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:35:56.969693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.961289 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0981d7f6-9826-4e4d-a643-09a2514967d0-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-p9dmt\" (UID: \"0981d7f6-9826-4e4d-a643-09a2514967d0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:35:56.969693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.961316 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0981d7f6-9826-4e4d-a643-09a2514967d0-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-p9dmt\" (UID: \"0981d7f6-9826-4e4d-a643-09a2514967d0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:35:56.969693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.961825 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0981d7f6-9826-4e4d-a643-09a2514967d0-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-p9dmt\" (UID: \"0981d7f6-9826-4e4d-a643-09a2514967d0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:35:56.973028 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.973000 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0981d7f6-9826-4e4d-a643-09a2514967d0-home\") pod \"echo-simulator-kserve-64b6f8f96c-p9dmt\" (UID: \"0981d7f6-9826-4e4d-a643-09a2514967d0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:35:56.985719 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.983994 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0981d7f6-9826-4e4d-a643-09a2514967d0-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-p9dmt\" (UID: \"0981d7f6-9826-4e4d-a643-09a2514967d0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:35:56.987488 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.987469 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0981d7f6-9826-4e4d-a643-09a2514967d0-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-p9dmt\" (UID: \"0981d7f6-9826-4e4d-a643-09a2514967d0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:35:56.989626 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:56.989604 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0981d7f6-9826-4e4d-a643-09a2514967d0-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-p9dmt\" (UID: \"0981d7f6-9826-4e4d-a643-09a2514967d0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:35:57.023483 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:57.023456 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8csx\" (UniqueName: \"kubernetes.io/projected/0981d7f6-9826-4e4d-a643-09a2514967d0-kube-api-access-n8csx\") pod \"echo-simulator-kserve-64b6f8f96c-p9dmt\" (UID: \"0981d7f6-9826-4e4d-a643-09a2514967d0\") " pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:35:57.056324 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:57.056303 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:35:57.455566 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:57.455544 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-p9dmt"] Nov 02 01:35:57.457591 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:35:57.457565 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0981d7f6_9826_4e4d_a643_09a2514967d0.slice/crio-ec246c631f359d73053dca19ada82ddfc7b157db5026888bcbabc19470ed0199 WatchSource:0}: Error finding container ec246c631f359d73053dca19ada82ddfc7b157db5026888bcbabc19470ed0199: Status 404 returned error can't find the container with id ec246c631f359d73053dca19ada82ddfc7b157db5026888bcbabc19470ed0199 Nov 02 01:35:57.760954 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:57.760924 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-w4b54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 02 01:35:57.761218 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:57.761189 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" podUID="832cc120-3c4e-4605-b990-683ca969b6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 02 01:35:58.436113 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:58.436084 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" event={"ID":"0981d7f6-9826-4e4d-a643-09a2514967d0","Type":"ContainerStarted","Data":"ec246c631f359d73053dca19ada82ddfc7b157db5026888bcbabc19470ed0199"} Nov 02 01:35:58.783512 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:58.783048 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-w4b54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 02 01:35:58.783512 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:58.783109 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" podUID="832cc120-3c4e-4605-b990-683ca969b6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 02 01:35:59.495663 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:59.495625 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:59.510086 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:59.510063 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:35:59.760568 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:59.760541 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-w4b54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 02 01:35:59.760745 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:35:59.760589 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" podUID="832cc120-3c4e-4605-b990-683ca969b6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 02 01:36:00.396461 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:00.396423 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:36:00.451445 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:00.451405 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" event={"ID":"0981d7f6-9826-4e4d-a643-09a2514967d0","Type":"ContainerStarted","Data":"0cb6818e79a192fbfa4d5b2aa3bf4999bc010a26f24fa755935e132241e85dfc"} Nov 02 01:36:00.463628 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:00.463605 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fg2fj" Nov 02 01:36:00.573532 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:00.573472 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" podStartSLOduration=1.805025082 podStartE2EDuration="4.573456687s" totalImagesPullingTime="2.768431605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:00.571996283 +0000 UTC m=+677.243248967" watchObservedRunningTime="2026-11-02 01:36:00.573456687 +0000 UTC m=+677.244709368" Nov 02 01:36:00.760369 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:00.760336 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-w4b54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 02 01:36:00.760549 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:00.760403 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" podUID="832cc120-3c4e-4605-b990-683ca969b6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 02 01:36:01.056929 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:01.056902 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:36:02.802845 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:02.802807 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-w4b54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:36:02.803403 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:02.802892 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" podUID="832cc120-3c4e-4605-b990-683ca969b6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:36:03.519337 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:03.519300 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:36:03.519686 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:03.519655 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:36:03.520386 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:03.520366 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:36:07.056980 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:07.056952 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:36:07.069521 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:07.069491 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:36:07.497200 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:07.497127 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-p9dmt" Nov 02 01:36:08.580760 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.580729 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr"] Nov 02 01:36:08.584524 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.584507 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.590258 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.590237 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.590367 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.590298 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.590470 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.590448 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.590516 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.590478 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.590516 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.590508 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.590612 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.590527 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.590612 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.590542 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.590700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.590637 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fltrt\" (UniqueName: \"kubernetes.io/projected/d13c6464-cb62-45eb-8cad-ca76333f3811-kube-api-access-fltrt\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.590784 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.590766 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d13c6464-cb62-45eb-8cad-ca76333f3811-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.685959 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.685879 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr"] Nov 02 01:36:08.691286 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.691257 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.691455 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.691322 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.691455 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.691358 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.691455 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.691381 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.691455 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.691430 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.691665 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.691457 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.691665 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.691479 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.691665 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.691509 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fltrt\" (UniqueName: \"kubernetes.io/projected/d13c6464-cb62-45eb-8cad-ca76333f3811-kube-api-access-fltrt\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.691665 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.691546 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d13c6464-cb62-45eb-8cad-ca76333f3811-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.691884 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.691745 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.692313 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.692278 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.692453 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.692317 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d13c6464-cb62-45eb-8cad-ca76333f3811-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.692910 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.692820 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.693169 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.693151 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.695544 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.695518 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.697190 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.697148 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.754063 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.754027 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.757124 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.757099 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fltrt\" (UniqueName: \"kubernetes.io/projected/d13c6464-cb62-45eb-8cad-ca76333f3811-kube-api-access-fltrt\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2qlpr\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:08.897916 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:08.897844 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:09.328994 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:09.328958 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr"] Nov 02 01:36:09.329747 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:36:09.329714 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13c6464_cb62_45eb_8cad_ca76333f3811.slice/crio-c54882d1c55844bdb540bbb9a96d75d43c79a0cb50bbe9b59311b87ff99ddb13 WatchSource:0}: Error finding container c54882d1c55844bdb540bbb9a96d75d43c79a0cb50bbe9b59311b87ff99ddb13: Status 404 returned error can't find the container with id c54882d1c55844bdb540bbb9a96d75d43c79a0cb50bbe9b59311b87ff99ddb13 Nov 02 01:36:09.332386 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:09.332349 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:36:09.332474 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:09.332426 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:36:09.332513 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:09.332493 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:36:09.502115 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:09.502078 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" event={"ID":"d13c6464-cb62-45eb-8cad-ca76333f3811","Type":"ContainerStarted","Data":"b8f61f11170ec304638a8e72f20f5519eaec6284a4e619c9060a3bab895b9a93"} Nov 02 01:36:09.502373 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:09.502350 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" event={"ID":"d13c6464-cb62-45eb-8cad-ca76333f3811","Type":"ContainerStarted","Data":"c54882d1c55844bdb540bbb9a96d75d43c79a0cb50bbe9b59311b87ff99ddb13"} Nov 02 01:36:09.639583 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:09.639464 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" podStartSLOduration=1.639445323 podStartE2EDuration="1.639445323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:09.625787094 +0000 UTC m=+686.297039778" watchObservedRunningTime="2026-11-02 01:36:09.639445323 +0000 UTC m=+686.310698008" Nov 02 01:36:09.899098 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:09.898996 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:10.899995 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:10.899962 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 02 01:36:10.900415 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:10.900022 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" podUID="d13c6464-cb62-45eb-8cad-ca76333f3811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 02 01:36:11.898250 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:11.898217 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 02 01:36:11.898435 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:11.898267 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" podUID="d13c6464-cb62-45eb-8cad-ca76333f3811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 02 01:36:12.898328 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:12.898292 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 02 01:36:12.898821 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:12.898356 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" podUID="d13c6464-cb62-45eb-8cad-ca76333f3811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 02 01:36:13.898924 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:13.898893 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 02 01:36:13.899326 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:13.898941 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" podUID="d13c6464-cb62-45eb-8cad-ca76333f3811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 02 01:36:14.898309 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:14.898269 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 02 01:36:14.898486 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:14.898332 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" podUID="d13c6464-cb62-45eb-8cad-ca76333f3811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 02 01:36:15.898225 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:15.898185 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 02 01:36:15.898757 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:15.898237 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" podUID="d13c6464-cb62-45eb-8cad-ca76333f3811" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 02 01:36:17.619574 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:17.619545 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:17.620075 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:17.619901 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:17.620638 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:17.620620 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:36:24.171988 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:36:24.171950 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:36:27.937210 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:27.937172 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5594cccc5c-rmdbn"] Nov 02 01:36:27.940823 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:27.940803 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5594cccc5c-rmdbn" Nov 02 01:36:27.940966 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:27.940934 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:36:28.045719 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:28.045664 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 01:36:28.045924 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:28.045832 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-f554h\"" Nov 02 01:36:28.046073 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:28.045920 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 01:36:28.048737 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:28.048710 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5594cccc5c-rmdbn"] Nov 02 01:36:28.070740 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:28.070711 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5d8c\" (UniqueName: \"kubernetes.io/projected/a1aecc70-488f-4ced-967e-9901f9a7e7d7-kube-api-access-c5d8c\") pod \"ai-gateway-controller-5594cccc5c-rmdbn\" (UID: \"a1aecc70-488f-4ced-967e-9901f9a7e7d7\") " pod="opendatahub/ai-gateway-controller-5594cccc5c-rmdbn" Nov 02 01:36:28.070740 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:28.070743 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1aecc70-488f-4ced-967e-9901f9a7e7d7-metrics-certs\") pod \"ai-gateway-controller-5594cccc5c-rmdbn\" (UID: \"a1aecc70-488f-4ced-967e-9901f9a7e7d7\") " pod="opendatahub/ai-gateway-controller-5594cccc5c-rmdbn" Nov 02 01:36:28.089371 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:28.089344 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:36:28.171911 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:28.171883 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5d8c\" (UniqueName: \"kubernetes.io/projected/a1aecc70-488f-4ced-967e-9901f9a7e7d7-kube-api-access-c5d8c\") pod \"ai-gateway-controller-5594cccc5c-rmdbn\" (UID: \"a1aecc70-488f-4ced-967e-9901f9a7e7d7\") " pod="opendatahub/ai-gateway-controller-5594cccc5c-rmdbn" Nov 02 01:36:28.172066 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:28.171918 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1aecc70-488f-4ced-967e-9901f9a7e7d7-metrics-certs\") pod \"ai-gateway-controller-5594cccc5c-rmdbn\" (UID: \"a1aecc70-488f-4ced-967e-9901f9a7e7d7\") " pod="opendatahub/ai-gateway-controller-5594cccc5c-rmdbn" Nov 02 01:36:28.174411 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:28.174387 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1aecc70-488f-4ced-967e-9901f9a7e7d7-metrics-certs\") pod \"ai-gateway-controller-5594cccc5c-rmdbn\" (UID: \"a1aecc70-488f-4ced-967e-9901f9a7e7d7\") " pod="opendatahub/ai-gateway-controller-5594cccc5c-rmdbn" Nov 02 01:36:28.233815 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:28.233761 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5d8c\" (UniqueName: \"kubernetes.io/projected/a1aecc70-488f-4ced-967e-9901f9a7e7d7-kube-api-access-c5d8c\") pod \"ai-gateway-controller-5594cccc5c-rmdbn\" (UID: \"a1aecc70-488f-4ced-967e-9901f9a7e7d7\") " pod="opendatahub/ai-gateway-controller-5594cccc5c-rmdbn" Nov 02 01:36:28.253588 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:28.253563 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5594cccc5c-rmdbn" Nov 02 01:36:28.623889 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:28.623865 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5594cccc5c-rmdbn"] Nov 02 01:36:28.625716 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:36:28.625685 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1aecc70_488f_4ced_967e_9901f9a7e7d7.slice/crio-2a8a05262e3828225c5a79e7e3566175a4327b363b4a85501f8a9bcea3fb6f78 WatchSource:0}: Error finding container 2a8a05262e3828225c5a79e7e3566175a4327b363b4a85501f8a9bcea3fb6f78: Status 404 returned error can't find the container with id 2a8a05262e3828225c5a79e7e3566175a4327b363b4a85501f8a9bcea3fb6f78 Nov 02 01:36:29.606973 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:29.606935 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5594cccc5c-rmdbn" event={"ID":"a1aecc70-488f-4ced-967e-9901f9a7e7d7","Type":"ContainerStarted","Data":"2a8a05262e3828225c5a79e7e3566175a4327b363b4a85501f8a9bcea3fb6f78"} Nov 02 01:36:34.637238 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:34.637203 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5594cccc5c-rmdbn" event={"ID":"a1aecc70-488f-4ced-967e-9901f9a7e7d7","Type":"ContainerStarted","Data":"c71ab3995efdcc26e856d594805e7b3330574fcee49a01ef393c94ea43c8d13e"} Nov 02 01:36:34.637655 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:34.637277 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5594cccc5c-rmdbn" Nov 02 01:36:34.756371 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:34.756322 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5594cccc5c-rmdbn" podStartSLOduration=1.824299632 podStartE2EDuration="7.756306598s" totalImagesPullingTime="5.932006966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:34.754499908 +0000 UTC m=+711.425752588" watchObservedRunningTime="2026-11-02 01:36:34.756306598 +0000 UTC m=+711.427559281" Nov 02 01:36:36.887334 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:36.887294 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:36:37.255575 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:37.255501 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:36:37.423966 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:37.423937 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:36:45.645048 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:45.645016 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5594cccc5c-rmdbn" Nov 02 01:36:46.007427 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:46.007334 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-684875949c-whjhb"] Nov 02 01:36:46.011282 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:46.011261 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-684875949c-whjhb" Nov 02 01:36:46.069839 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:46.069810 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-684875949c-whjhb"] Nov 02 01:36:46.135121 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:46.135090 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a9d8c11-36a4-4fce-828e-86089d79dba4-metrics-certs\") pod \"ai-gateway-controller-684875949c-whjhb\" (UID: \"2a9d8c11-36a4-4fce-828e-86089d79dba4\") " pod="opendatahub/ai-gateway-controller-684875949c-whjhb" Nov 02 01:36:46.135237 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:46.135206 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6twvh\" (UniqueName: \"kubernetes.io/projected/2a9d8c11-36a4-4fce-828e-86089d79dba4-kube-api-access-6twvh\") pod \"ai-gateway-controller-684875949c-whjhb\" (UID: \"2a9d8c11-36a4-4fce-828e-86089d79dba4\") " pod="opendatahub/ai-gateway-controller-684875949c-whjhb" Nov 02 01:36:46.236090 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:46.236038 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6twvh\" (UniqueName: \"kubernetes.io/projected/2a9d8c11-36a4-4fce-828e-86089d79dba4-kube-api-access-6twvh\") pod \"ai-gateway-controller-684875949c-whjhb\" (UID: \"2a9d8c11-36a4-4fce-828e-86089d79dba4\") " pod="opendatahub/ai-gateway-controller-684875949c-whjhb" Nov 02 01:36:46.236284 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:46.236141 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a9d8c11-36a4-4fce-828e-86089d79dba4-metrics-certs\") pod \"ai-gateway-controller-684875949c-whjhb\" (UID: \"2a9d8c11-36a4-4fce-828e-86089d79dba4\") " pod="opendatahub/ai-gateway-controller-684875949c-whjhb" Nov 02 01:36:46.238616 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:46.238586 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a9d8c11-36a4-4fce-828e-86089d79dba4-metrics-certs\") pod \"ai-gateway-controller-684875949c-whjhb\" (UID: \"2a9d8c11-36a4-4fce-828e-86089d79dba4\") " pod="opendatahub/ai-gateway-controller-684875949c-whjhb" Nov 02 01:36:46.298171 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:46.298138 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6twvh\" (UniqueName: \"kubernetes.io/projected/2a9d8c11-36a4-4fce-828e-86089d79dba4-kube-api-access-6twvh\") pod \"ai-gateway-controller-684875949c-whjhb\" (UID: \"2a9d8c11-36a4-4fce-828e-86089d79dba4\") " pod="opendatahub/ai-gateway-controller-684875949c-whjhb" Nov 02 01:36:46.323623 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:46.323591 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-684875949c-whjhb" Nov 02 01:36:46.695415 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:46.695392 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-684875949c-whjhb"] Nov 02 01:36:46.697455 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:36:46.697430 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9d8c11_36a4_4fce_828e_86089d79dba4.slice/crio-2bf0a1436ce80bfeab5ab6252415aac55fee524767e4b2fafb5808e9b95a00b6 WatchSource:0}: Error finding container 2bf0a1436ce80bfeab5ab6252415aac55fee524767e4b2fafb5808e9b95a00b6: Status 404 returned error can't find the container with id 2bf0a1436ce80bfeab5ab6252415aac55fee524767e4b2fafb5808e9b95a00b6 Nov 02 01:36:46.707740 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:46.707707 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-684875949c-whjhb" event={"ID":"2a9d8c11-36a4-4fce-828e-86089d79dba4","Type":"ContainerStarted","Data":"2bf0a1436ce80bfeab5ab6252415aac55fee524767e4b2fafb5808e9b95a00b6"} Nov 02 01:36:47.714209 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:47.714177 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-684875949c-whjhb" event={"ID":"2a9d8c11-36a4-4fce-828e-86089d79dba4","Type":"ContainerStarted","Data":"2911340e499d38148edc2bf230733698d6ad3a3d1d55fa34173ad191fffd2957"} Nov 02 01:36:47.714572 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:47.714294 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-684875949c-whjhb" Nov 02 01:36:47.832865 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:47.832771 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-684875949c-whjhb" podStartSLOduration=1.9805055139999999 podStartE2EDuration="2.832756368s" totalImagesPullingTime="852.250854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:47.831281974 +0000 UTC m=+724.502534655" watchObservedRunningTime="2026-11-02 01:36:47.832756368 +0000 UTC m=+724.504009100" Nov 02 01:36:53.950969 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:53.950937 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps"] Nov 02 01:36:53.954708 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:53.954687 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.061657 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.061623 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps"] Nov 02 01:36:54.109867 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.109775 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.110025 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.109902 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.110025 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.109951 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.110101 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.110030 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.110101 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.110058 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.110101 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.110097 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.110196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.110155 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.110229 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.110197 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82fc1a10-7f06-4790-94bd-c57c06c06803-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.110229 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.110217 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn9c4\" (UniqueName: \"kubernetes.io/projected/82fc1a10-7f06-4790-94bd-c57c06c06803-kube-api-access-qn9c4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.172023 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:36:54.171987 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:36:54.211572 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.211510 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.211572 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.211549 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.211572 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.211569 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82fc1a10-7f06-4790-94bd-c57c06c06803-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.211835 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.211694 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn9c4\" (UniqueName: \"kubernetes.io/projected/82fc1a10-7f06-4790-94bd-c57c06c06803-kube-api-access-qn9c4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.211835 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.211735 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.211835 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.211762 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.211835 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.211806 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.212065 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.212037 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.212125 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.212080 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.212197 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.212176 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.212303 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.212274 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.212431 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.212389 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.212431 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.212417 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.212547 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.212480 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82fc1a10-7f06-4790-94bd-c57c06c06803-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.213778 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.213756 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.214112 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.214095 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.273418 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.273395 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn9c4\" (UniqueName: \"kubernetes.io/projected/82fc1a10-7f06-4790-94bd-c57c06c06803-kube-api-access-qn9c4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.273508 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.273393 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-cj4ps\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.567283 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.567254 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:54.993744 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.993718 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps"] Nov 02 01:36:54.996136 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:36:54.996097 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82fc1a10_7f06_4790_94bd_c57c06c06803.slice/crio-69420e77b9e0f4f42858ebc036fab8602c122d9d0b21bfb450d8952d5830dce7 WatchSource:0}: Error finding container 69420e77b9e0f4f42858ebc036fab8602c122d9d0b21bfb450d8952d5830dce7: Status 404 returned error can't find the container with id 69420e77b9e0f4f42858ebc036fab8602c122d9d0b21bfb450d8952d5830dce7 Nov 02 01:36:54.998332 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.998304 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:36:54.998393 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.998364 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:36:54.998429 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:54.998394 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:36:55.754287 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:55.754244 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" event={"ID":"82fc1a10-7f06-4790-94bd-c57c06c06803","Type":"ContainerStarted","Data":"d47aea04e42b19bd5c06e77bc6c9d48280cb032c75df7f1b668cba60d7030155"} Nov 02 01:36:55.754507 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:55.754310 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" event={"ID":"82fc1a10-7f06-4790-94bd-c57c06c06803","Type":"ContainerStarted","Data":"69420e77b9e0f4f42858ebc036fab8602c122d9d0b21bfb450d8952d5830dce7"} Nov 02 01:36:55.927646 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:55.927596 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" podStartSLOduration=2.9275823279999997 podStartE2EDuration="2.927582328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:36:55.926941186 +0000 UTC m=+732.598193867" watchObservedRunningTime="2026-11-02 01:36:55.927582328 +0000 UTC m=+732.598835009" Nov 02 01:36:56.583695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:56.578919 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:36:56.586475 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:56.586451 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 02 01:36:56.586588 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:56.586496 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" podUID="82fc1a10-7f06-4790-94bd-c57c06c06803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 02 01:36:57.537887 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:57.537843 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:36:57.581694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:57.574970 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 02 01:36:57.581694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:57.575021 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" podUID="82fc1a10-7f06-4790-94bd-c57c06c06803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 02 01:36:57.688267 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:57.688234 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:36:58.581695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:58.577018 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 02 01:36:58.581695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:58.577080 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" podUID="82fc1a10-7f06-4790-94bd-c57c06c06803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 02 01:36:58.721517 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:58.721489 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-684875949c-whjhb" Nov 02 01:36:58.893169 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:58.891176 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5594cccc5c-rmdbn"] Nov 02 01:36:58.893169 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:58.891830 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5594cccc5c-rmdbn" podUID="a1aecc70-488f-4ced-967e-9901f9a7e7d7" containerName="manager" containerID="cri-o://c71ab3995efdcc26e856d594805e7b3330574fcee49a01ef393c94ea43c8d13e" gracePeriod=30 Nov 02 01:36:59.194165 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:59.194138 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5594cccc5c-rmdbn" Nov 02 01:36:59.258012 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:59.257983 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1aecc70-488f-4ced-967e-9901f9a7e7d7-kube-api-access-c5d8c\" (UniqueName: \"kubernetes.io/projected/a1aecc70-488f-4ced-967e-9901f9a7e7d7-kube-api-access-c5d8c\") pod \"a1aecc70-488f-4ced-967e-9901f9a7e7d7\" (UID: \"a1aecc70-488f-4ced-967e-9901f9a7e7d7\") " Nov 02 01:36:59.258157 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:59.258017 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1aecc70-488f-4ced-967e-9901f9a7e7d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1aecc70-488f-4ced-967e-9901f9a7e7d7-metrics-certs\") pod \"a1aecc70-488f-4ced-967e-9901f9a7e7d7\" (UID: \"a1aecc70-488f-4ced-967e-9901f9a7e7d7\") " Nov 02 01:36:59.260065 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:59.260040 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1aecc70-488f-4ced-967e-9901f9a7e7d7-kube-api-access-c5d8c" pod "a1aecc70-488f-4ced-967e-9901f9a7e7d7" (UID: "a1aecc70-488f-4ced-967e-9901f9a7e7d7"). InnerVolumeSpecName "kube-api-access-c5d8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:36:59.260185 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:59.260080 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1aecc70-488f-4ced-967e-9901f9a7e7d7-metrics-certs" pod "a1aecc70-488f-4ced-967e-9901f9a7e7d7" (UID: "a1aecc70-488f-4ced-967e-9901f9a7e7d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:36:59.358712 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:59.358655 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5d8c\" (UniqueName: \"kubernetes.io/projected/a1aecc70-488f-4ced-967e-9901f9a7e7d7-kube-api-access-c5d8c\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:36:59.358712 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:59.358708 5044 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1aecc70-488f-4ced-967e-9901f9a7e7d7-metrics-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:36:59.582694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:59.581949 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 02 01:36:59.582694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:59.582004 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" podUID="82fc1a10-7f06-4790-94bd-c57c06c06803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 02 01:36:59.799311 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:59.799280 5044 generic.go:356] "Generic (PLEG): container finished" podID="a1aecc70-488f-4ced-967e-9901f9a7e7d7" containerID="c71ab3995efdcc26e856d594805e7b3330574fcee49a01ef393c94ea43c8d13e" exitCode=0 Nov 02 01:36:59.799769 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:59.799324 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5594cccc5c-rmdbn" event={"ID":"a1aecc70-488f-4ced-967e-9901f9a7e7d7","Type":"ContainerDied","Data":"c71ab3995efdcc26e856d594805e7b3330574fcee49a01ef393c94ea43c8d13e"} Nov 02 01:36:59.799769 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:59.799332 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5594cccc5c-rmdbn" Nov 02 01:36:59.799769 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:59.799365 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5594cccc5c-rmdbn" event={"ID":"a1aecc70-488f-4ced-967e-9901f9a7e7d7","Type":"ContainerDied","Data":"2a8a05262e3828225c5a79e7e3566175a4327b363b4a85501f8a9bcea3fb6f78"} Nov 02 01:36:59.799769 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:59.799386 5044 scope.go:122] "RemoveContainer" containerID="c71ab3995efdcc26e856d594805e7b3330574fcee49a01ef393c94ea43c8d13e" Nov 02 01:36:59.807986 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:59.807967 5044 scope.go:122] "RemoveContainer" containerID="c71ab3995efdcc26e856d594805e7b3330574fcee49a01ef393c94ea43c8d13e" Nov 02 01:36:59.808284 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:36:59.808263 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71ab3995efdcc26e856d594805e7b3330574fcee49a01ef393c94ea43c8d13e\": container with ID starting with c71ab3995efdcc26e856d594805e7b3330574fcee49a01ef393c94ea43c8d13e not found: ID does not exist" containerID="c71ab3995efdcc26e856d594805e7b3330574fcee49a01ef393c94ea43c8d13e" Nov 02 01:36:59.808343 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:59.808292 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71ab3995efdcc26e856d594805e7b3330574fcee49a01ef393c94ea43c8d13e"} err="failed to get container status \"c71ab3995efdcc26e856d594805e7b3330574fcee49a01ef393c94ea43c8d13e\": rpc error: code = NotFound desc = could not find container \"c71ab3995efdcc26e856d594805e7b3330574fcee49a01ef393c94ea43c8d13e\": container with ID starting with c71ab3995efdcc26e856d594805e7b3330574fcee49a01ef393c94ea43c8d13e not found: ID does not exist" Nov 02 01:36:59.981109 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:36:59.981077 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5594cccc5c-rmdbn"] Nov 02 01:37:00.030725 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:00.030693 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5594cccc5c-rmdbn"] Nov 02 01:37:00.585860 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:00.585820 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 02 01:37:00.586060 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:00.585892 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" podUID="82fc1a10-7f06-4790-94bd-c57c06c06803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 02 01:37:00.586829 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:00.586801 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:01.144227 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:01.144198 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:01.451816 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:01.451722 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1aecc70-488f-4ced-967e-9901f9a7e7d7" path="/var/lib/kubelet/pods/a1aecc70-488f-4ced-967e-9901f9a7e7d7/volumes" Nov 02 01:37:01.568643 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:01.568608 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 02 01:37:01.568819 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:01.568732 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" podUID="82fc1a10-7f06-4790-94bd-c57c06c06803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 02 01:37:02.779115 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:02.779079 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:02.926724 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:02.926683 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:03.229140 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:03.229047 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:37:03.229475 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:03.229451 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:37:03.230327 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:03.230309 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:37:04.614887 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:04.614857 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:04.761053 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:04.761020 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:24.171713 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:37:24.171602 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:37:33.841336 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:33.841293 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:33.990710 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:33.990666 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:34.378930 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.378897 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv"] Nov 02 01:37:34.379329 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.379317 5044 cpu_manager.go:414] "containerMap: removing container" podUID="a1aecc70-488f-4ced-967e-9901f9a7e7d7" containerName="manager" Nov 02 01:37:34.379373 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.379331 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1aecc70-488f-4ced-967e-9901f9a7e7d7" containerName="manager" Nov 02 01:37:34.379431 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.379422 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1aecc70-488f-4ced-967e-9901f9a7e7d7" containerName="manager" Nov 02 01:37:34.382656 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.382636 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.436763 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.436730 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-f276c3-openshift-default-dockercfg-72fq4\"" Nov 02 01:37:34.469214 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.469184 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:34.488861 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.488833 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv"] Nov 02 01:37:34.500716 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.500691 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6zj8\" (UniqueName: \"kubernetes.io/projected/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-kube-api-access-l6zj8\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.500850 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.500727 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.500850 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.500745 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.500850 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.500762 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.500998 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.500919 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.501040 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.501016 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.501081 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.501044 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.501081 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.501067 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.501198 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.501182 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.602570 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.602542 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6zj8\" (UniqueName: \"kubernetes.io/projected/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-kube-api-access-l6zj8\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.602741 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.602576 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.602741 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.602592 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.602741 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.602609 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.602741 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.602650 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.602961 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.602796 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.602961 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.602833 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.602961 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.602879 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.602961 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.602959 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.603168 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.603067 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.603168 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.603119 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.603265 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.603202 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.603385 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.603362 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.603690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.603648 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.605500 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.605481 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.605650 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.605633 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.665798 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.665725 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.665908 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.665852 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6zj8\" (UniqueName: \"kubernetes.io/projected/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-kube-api-access-l6zj8\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.672026 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.672004 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:34.696004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.695978 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:34.817967 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:34.817919 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:35.098240 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.098200 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:35.157073 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.157052 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv"] Nov 02 01:37:35.158634 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:37:35.158598 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1af3dd7_0ab8_4feb_a39c_7e15f9d2c7fa.slice/crio-a4ec4635ef576fd67611c9858b531e316ee30072ab07b1b6c50e91441b5adfa3 WatchSource:0}: Error finding container a4ec4635ef576fd67611c9858b531e316ee30072ab07b1b6c50e91441b5adfa3: Status 404 returned error can't find the container with id a4ec4635ef576fd67611c9858b531e316ee30072ab07b1b6c50e91441b5adfa3 Nov 02 01:37:35.160868 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.160837 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:37:35.160971 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.160903 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:37:35.160971 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.160935 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:37:35.253619 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.253595 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:35.316851 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.316820 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657"] Nov 02 01:37:35.320479 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.320458 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.374131 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.374052 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5888dc-openshift-default-dockercfg-cgmjh\"" Nov 02 01:37:35.384729 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.384699 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657"] Nov 02 01:37:35.395237 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.395208 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:35.410143 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.410114 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.410293 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.410164 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.410362 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.410284 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.410362 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.410326 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.410474 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.410360 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.410474 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.410409 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.410474 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.410453 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.410624 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.410509 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.410624 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.410540 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncwsw\" (UniqueName: \"kubernetes.io/projected/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-kube-api-access-ncwsw\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.511191 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.511161 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.511343 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.511202 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.511343 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.511263 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.511343 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.511295 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncwsw\" (UniqueName: \"kubernetes.io/projected/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-kube-api-access-ncwsw\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.511484 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.511392 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.511484 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.511431 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.511583 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.511481 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.511583 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.511516 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.511583 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.511549 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.511914 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.511891 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.512005 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.511924 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.512005 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.511997 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.512179 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.512153 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.512179 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.512172 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.513924 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.513898 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.514085 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.514070 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.538858 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.538831 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:35.573491 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.573464 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.573631 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.573516 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncwsw\" (UniqueName: \"kubernetes.io/projected/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-kube-api-access-ncwsw\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.634525 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.634453 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:35.670113 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.670080 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj"] Nov 02 01:37:35.674397 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.674379 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.704694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.704589 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:35.727966 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.727935 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-e06ea2-openshift-default-dockercfg-x8mvt\"" Nov 02 01:37:35.735322 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.734712 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj"] Nov 02 01:37:35.815826 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.815787 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.816035 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.815841 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.816035 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.815975 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.816035 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.816018 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.816334 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.816060 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.816334 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.816093 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c8lq\" (UniqueName: \"kubernetes.io/projected/652e50af-8a91-43ff-bb49-bbdf781feefd-kube-api-access-8c8lq\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.816334 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.816137 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.816334 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.816176 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652e50af-8a91-43ff-bb49-bbdf781feefd-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.816334 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.816244 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.848336 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.848308 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:35.917291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.917224 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652e50af-8a91-43ff-bb49-bbdf781feefd-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.917419 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.917313 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.917419 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.917364 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.917419 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.917396 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.917563 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.917441 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.917563 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.917473 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.917563 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.917514 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.917563 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.917546 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c8lq\" (UniqueName: \"kubernetes.io/projected/652e50af-8a91-43ff-bb49-bbdf781feefd-kube-api-access-8c8lq\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.917784 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.917591 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.917897 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.917877 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.918075 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.918052 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.918150 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.918083 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.918150 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.918106 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.918367 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.918346 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652e50af-8a91-43ff-bb49-bbdf781feefd-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.920773 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.920752 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.920866 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.920852 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.983890 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.983857 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" event={"ID":"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa","Type":"ContainerStarted","Data":"cd4da6ec8f1b51383423c155fb319292fa6c7bf32aa9b04bf546d2bccb7655b7"} Nov 02 01:37:35.984087 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.983920 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" event={"ID":"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa","Type":"ContainerStarted","Data":"a4ec4635ef576fd67611c9858b531e316ee30072ab07b1b6c50e91441b5adfa3"} Nov 02 01:37:35.985848 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.985822 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:35.987597 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:35.987574 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c8lq\" (UniqueName: \"kubernetes.io/projected/652e50af-8a91-43ff-bb49-bbdf781feefd-kube-api-access-8c8lq\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:36.020062 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.020028 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657"] Nov 02 01:37:36.022788 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.022524 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:36.025243 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.025209 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:36.031504 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:37:36.031467 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d46753_0b7a_4c8c_9870_31cedadbe4c5.slice/crio-21314af556ab56dcc6833169de06ae0a1489f288701ed48cce35b4deaa7f67d2 WatchSource:0}: Error finding container 21314af556ab56dcc6833169de06ae0a1489f288701ed48cce35b4deaa7f67d2: Status 404 returned error can't find the container with id 21314af556ab56dcc6833169de06ae0a1489f288701ed48cce35b4deaa7f67d2 Nov 02 01:37:36.034360 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.034321 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:37:36.034465 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.034406 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:37:36.034525 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.034474 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:37:36.125379 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.124991 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" podStartSLOduration=2.124972446 podStartE2EDuration="2.124972446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:36.119084478 +0000 UTC m=+772.790337162" watchObservedRunningTime="2026-11-02 01:37:36.124972446 +0000 UTC m=+772.796225130" Nov 02 01:37:36.259737 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.259699 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:36.410012 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.409973 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:36.430955 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.430931 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj"] Nov 02 01:37:36.432548 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:37:36.432519 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652e50af_8a91_43ff_bb49_bbdf781feefd.slice/crio-c1314c4b60307e69af2fcc8e3ff2fa8f5c983de81d52c1149f9058931924a5f3 WatchSource:0}: Error finding container c1314c4b60307e69af2fcc8e3ff2fa8f5c983de81d52c1149f9058931924a5f3: Status 404 returned error can't find the container with id c1314c4b60307e69af2fcc8e3ff2fa8f5c983de81d52c1149f9058931924a5f3 Nov 02 01:37:36.435351 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.435317 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:37:36.435420 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.435394 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:37:36.435472 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.435435 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:37:36.561741 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.561643 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:36.696657 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.696383 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:36.704342 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.704311 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:36.710742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.710718 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:36.991195 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.991111 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" event={"ID":"b7d46753-0b7a-4c8c-9870-31cedadbe4c5","Type":"ContainerStarted","Data":"b490bbb8a5b53dfe58e293e00dcb367abc4b391097731dc7baa85a5acc11329b"} Nov 02 01:37:36.991195 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.991172 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" event={"ID":"b7d46753-0b7a-4c8c-9870-31cedadbe4c5","Type":"ContainerStarted","Data":"21314af556ab56dcc6833169de06ae0a1489f288701ed48cce35b4deaa7f67d2"} Nov 02 01:37:36.993232 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.993199 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" event={"ID":"652e50af-8a91-43ff-bb49-bbdf781feefd","Type":"ContainerStarted","Data":"acd61ca1e138ca51879c99ddcee190bd2c8d06866d65c415320e2218eb3130a3"} Nov 02 01:37:36.993380 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.993254 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" event={"ID":"652e50af-8a91-43ff-bb49-bbdf781feefd","Type":"ContainerStarted","Data":"c1314c4b60307e69af2fcc8e3ff2fa8f5c983de81d52c1149f9058931924a5f3"} Nov 02 01:37:36.993450 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.993436 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:36.994378 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:36.994358 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:37:37.023608 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:37.023585 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:37.028230 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:37.028211 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:37.116375 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:37.116304 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" podStartSLOduration=2.116291606 podStartE2EDuration="2.116291606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:37.115309918 +0000 UTC m=+773.786562837" watchObservedRunningTime="2026-11-02 01:37:37.116291606 +0000 UTC m=+773.787544286" Nov 02 01:37:37.225486 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:37.225457 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:37.365867 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:37.365817 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" podStartSLOduration=2.365803174 podStartE2EDuration="2.365803174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:37:37.365182302 +0000 UTC m=+774.036434983" watchObservedRunningTime="2026-11-02 01:37:37.365803174 +0000 UTC m=+774.037055848" Nov 02 01:37:37.375015 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:37.374986 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:37.638153 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:37.638019 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:37.646004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:37.645971 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:37.999023 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:37.998935 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:37.999023 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:37.998999 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:37.999651 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:37.999616 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:37:37.999941 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:37.999913 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:37:39.413941 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:39.413910 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:39.595774 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:39.595737 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:39.740062 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:39.739977 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:40.211953 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:40.211923 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:40.421716 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:40.421662 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:40.562777 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:40.562742 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:40.869903 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:40.869830 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:41.079240 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:41.079171 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:41.227277 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:41.227204 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:41.593395 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:41.593356 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:41.784929 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:41.784899 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:41.934503 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:41.934429 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:42.251636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:42.251565 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:42.481861 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:42.481828 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:42.622883 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:42.622851 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:42.943194 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:42.943113 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:43.152765 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:43.152731 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:43.299032 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:43.299005 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:44.657783 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:44.657748 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:44.894577 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:44.894541 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:45.036404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:45.036373 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:50.757012 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:50.756981 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:50.913039 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:50.913007 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:51.053503 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:51.053469 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:54.172184 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:37:54.172150 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:37:56.019349 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:56.019312 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:56.175585 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:56.175557 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:56.320967 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:56.320885 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:56.628059 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:56.627980 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:56.801051 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:56.801017 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:37:57.200214 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:57.200178 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr"] Nov 02 01:37:57.211016 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:57.210986 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr" Nov 02 01:37:57.256366 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:57.256340 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr"] Nov 02 01:37:57.304542 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:57.304518 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-e06ea2\"" Nov 02 01:37:57.304701 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:57.304591 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-e06ea2-dockercfg-sbglj\"" Nov 02 01:37:57.331004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:57.330975 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm4m5\" (UniqueName: \"kubernetes.io/projected/152ed332-7e6a-4864-9763-726b003e585e-kube-api-access-rm4m5\") pod \"payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr\" (UID: \"152ed332-7e6a-4864-9763-726b003e585e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr" Nov 02 01:37:57.331135 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:57.331022 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/152ed332-7e6a-4864-9763-726b003e585e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr\" (UID: \"152ed332-7e6a-4864-9763-726b003e585e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr" Nov 02 01:37:57.432026 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:57.431992 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm4m5\" (UniqueName: \"kubernetes.io/projected/152ed332-7e6a-4864-9763-726b003e585e-kube-api-access-rm4m5\") pod \"payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr\" (UID: \"152ed332-7e6a-4864-9763-726b003e585e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr" Nov 02 01:37:57.432202 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:57.432050 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/152ed332-7e6a-4864-9763-726b003e585e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr\" (UID: \"152ed332-7e6a-4864-9763-726b003e585e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr" Nov 02 01:37:57.433104 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:57.433078 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/152ed332-7e6a-4864-9763-726b003e585e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr\" (UID: \"152ed332-7e6a-4864-9763-726b003e585e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr" Nov 02 01:37:57.494101 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:57.494033 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm4m5\" (UniqueName: \"kubernetes.io/projected/152ed332-7e6a-4864-9763-726b003e585e-kube-api-access-rm4m5\") pod \"payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr\" (UID: \"152ed332-7e6a-4864-9763-726b003e585e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr" Nov 02 01:37:57.525115 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:57.525087 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr" Nov 02 01:37:57.895324 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:57.895300 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr"] Nov 02 01:37:57.896879 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:37:57.896855 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod152ed332_7e6a_4864_9763_726b003e585e.slice/crio-04c87fa084ee88e40fe09812fda8090341f60966723fc3047d240e75bab66486 WatchSource:0}: Error finding container 04c87fa084ee88e40fe09812fda8090341f60966723fc3047d240e75bab66486: Status 404 returned error can't find the container with id 04c87fa084ee88e40fe09812fda8090341f60966723fc3047d240e75bab66486 Nov 02 01:37:58.102755 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:37:58.102713 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr" event={"ID":"152ed332-7e6a-4864-9763-726b003e585e","Type":"ContainerStarted","Data":"04c87fa084ee88e40fe09812fda8090341f60966723fc3047d240e75bab66486"} Nov 02 01:38:00.912422 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:00.912395 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:01.122840 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:01.122804 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:01.132131 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:01.132085 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr" event={"ID":"152ed332-7e6a-4864-9763-726b003e585e","Type":"ContainerStarted","Data":"f63264e0afe1e9c15d2203a9fc932414d64904c16919ad8f8d2e1de7e3af7c7f"} Nov 02 01:38:01.132328 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:01.132156 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr" Nov 02 01:38:01.254388 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:01.254293 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr" podStartSLOduration=1.24766243 podStartE2EDuration="4.254279203s" totalImagesPullingTime="3.006616773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:01.252769997 +0000 UTC m=+797.924022679" watchObservedRunningTime="2026-11-02 01:38:01.254279203 +0000 UTC m=+797.925531934" Nov 02 01:38:01.282234 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:01.282201 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:01.686367 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:01.686312 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:01.848030 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:01.848001 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:02.090756 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:02.090723 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9"] Nov 02 01:38:02.094448 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:02.094432 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9" Nov 02 01:38:02.149174 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:02.149142 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-36a1e0\"" Nov 02 01:38:02.151819 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:02.151795 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9"] Nov 02 01:38:02.208768 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:02.208730 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jsl9\" (UniqueName: \"kubernetes.io/projected/bd419016-7531-4bcc-9f84-e07ee7a5d747-kube-api-access-7jsl9\") pod \"payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9\" (UID: \"bd419016-7531-4bcc-9f84-e07ee7a5d747\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9" Nov 02 01:38:02.208768 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:02.208773 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd419016-7531-4bcc-9f84-e07ee7a5d747-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9\" (UID: \"bd419016-7531-4bcc-9f84-e07ee7a5d747\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9" Nov 02 01:38:02.313708 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:02.310608 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jsl9\" (UniqueName: \"kubernetes.io/projected/bd419016-7531-4bcc-9f84-e07ee7a5d747-kube-api-access-7jsl9\") pod \"payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9\" (UID: \"bd419016-7531-4bcc-9f84-e07ee7a5d747\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9" Nov 02 01:38:02.313708 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:02.310644 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd419016-7531-4bcc-9f84-e07ee7a5d747-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9\" (UID: \"bd419016-7531-4bcc-9f84-e07ee7a5d747\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9" Nov 02 01:38:02.313708 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:02.311406 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd419016-7531-4bcc-9f84-e07ee7a5d747-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9\" (UID: \"bd419016-7531-4bcc-9f84-e07ee7a5d747\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9" Nov 02 01:38:02.393662 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:02.393590 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jsl9\" (UniqueName: \"kubernetes.io/projected/bd419016-7531-4bcc-9f84-e07ee7a5d747-kube-api-access-7jsl9\") pod \"payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9\" (UID: \"bd419016-7531-4bcc-9f84-e07ee7a5d747\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9" Nov 02 01:38:02.407631 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:02.407596 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9" Nov 02 01:38:02.797237 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:02.797213 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9"] Nov 02 01:38:02.799167 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:38:02.799131 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd419016_7531_4bcc_9f84_e07ee7a5d747.slice/crio-1fea021420fcb9e41389fb012998912abf9049941e3f1017fa85bcfa85f88083 WatchSource:0}: Error finding container 1fea021420fcb9e41389fb012998912abf9049941e3f1017fa85bcfa85f88083: Status 404 returned error can't find the container with id 1fea021420fcb9e41389fb012998912abf9049941e3f1017fa85bcfa85f88083 Nov 02 01:38:03.146866 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:03.146785 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9" event={"ID":"bd419016-7531-4bcc-9f84-e07ee7a5d747","Type":"ContainerStarted","Data":"2f5e639657b2985439ad95cb4395ac336e743645325739fb6f58c0768b408865"} Nov 02 01:38:03.146866 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:03.146844 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9" event={"ID":"bd419016-7531-4bcc-9f84-e07ee7a5d747","Type":"ContainerStarted","Data":"1fea021420fcb9e41389fb012998912abf9049941e3f1017fa85bcfa85f88083"} Nov 02 01:38:03.147296 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:03.146928 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9" Nov 02 01:38:03.270693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:03.266584 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9" podStartSLOduration=1.266565556 podStartE2EDuration="1.266565556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:03.265076675 +0000 UTC m=+799.936329359" watchObservedRunningTime="2026-11-02 01:38:03.266565556 +0000 UTC m=+799.937818239" Nov 02 01:38:05.818533 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:05.818497 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:05.973895 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:05.973866 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:06.151951 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:06.151844 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:06.539496 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:06.539459 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:06.710922 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:06.710881 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:07.098657 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:07.098614 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2"] Nov 02 01:38:07.104823 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:07.104799 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2" Nov 02 01:38:07.159119 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:07.159084 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2"] Nov 02 01:38:07.165351 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:07.164954 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfeb96d5-3e36-4855-b2e6-7cf5311d131b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2\" (UID: \"cfeb96d5-3e36-4855-b2e6-7cf5311d131b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2" Nov 02 01:38:07.165351 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:07.165048 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd8nt\" (UniqueName: \"kubernetes.io/projected/cfeb96d5-3e36-4855-b2e6-7cf5311d131b-kube-api-access-pd8nt\") pod \"payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2\" (UID: \"cfeb96d5-3e36-4855-b2e6-7cf5311d131b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2" Nov 02 01:38:07.206772 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:07.206742 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-f276c3\"" Nov 02 01:38:07.207052 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:07.207025 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-f276c3-dockercfg-kjpcl\"" Nov 02 01:38:07.266221 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:07.266185 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfeb96d5-3e36-4855-b2e6-7cf5311d131b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2\" (UID: \"cfeb96d5-3e36-4855-b2e6-7cf5311d131b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2" Nov 02 01:38:07.266387 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:07.266239 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd8nt\" (UniqueName: \"kubernetes.io/projected/cfeb96d5-3e36-4855-b2e6-7cf5311d131b-kube-api-access-pd8nt\") pod \"payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2\" (UID: \"cfeb96d5-3e36-4855-b2e6-7cf5311d131b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2" Nov 02 01:38:07.267007 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:07.266983 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfeb96d5-3e36-4855-b2e6-7cf5311d131b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2\" (UID: \"cfeb96d5-3e36-4855-b2e6-7cf5311d131b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2" Nov 02 01:38:07.328548 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:07.328509 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd8nt\" (UniqueName: \"kubernetes.io/projected/cfeb96d5-3e36-4855-b2e6-7cf5311d131b-kube-api-access-pd8nt\") pod \"payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2\" (UID: \"cfeb96d5-3e36-4855-b2e6-7cf5311d131b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2" Nov 02 01:38:07.422269 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:07.422195 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2" Nov 02 01:38:07.817544 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:07.817503 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2"] Nov 02 01:38:07.818487 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:38:07.818458 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfeb96d5_3e36_4855_b2e6_7cf5311d131b.slice/crio-c8427eec97af96388c52f1ff5581319cc14a211fc5e736b88b2a66df4d80fd05 WatchSource:0}: Error finding container c8427eec97af96388c52f1ff5581319cc14a211fc5e736b88b2a66df4d80fd05: Status 404 returned error can't find the container with id c8427eec97af96388c52f1ff5581319cc14a211fc5e736b88b2a66df4d80fd05 Nov 02 01:38:08.179033 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:08.178927 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2" event={"ID":"cfeb96d5-3e36-4855-b2e6-7cf5311d131b","Type":"ContainerStarted","Data":"ec0fe4eab8143f04a4560c06c856130318149f1fa4cb028c9087632d10653123"} Nov 02 01:38:08.179033 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:08.178998 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2" event={"ID":"cfeb96d5-3e36-4855-b2e6-7cf5311d131b","Type":"ContainerStarted","Data":"c8427eec97af96388c52f1ff5581319cc14a211fc5e736b88b2a66df4d80fd05"} Nov 02 01:38:08.179033 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:08.179031 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2" Nov 02 01:38:08.276626 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:08.276579 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": read tcp 10.132.0.2:60446->10.132.0.50:15021: read: connection reset by peer" start-of-body= Nov 02 01:38:08.276948 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:08.276644 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" podUID="652e50af-8a91-43ff-bb49-bbdf781feefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": read tcp 10.132.0.2:60446->10.132.0.50:15021: read: connection reset by peer" Nov 02 01:38:08.350683 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:08.350623 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2" podStartSLOduration=1.350609003 podStartE2EDuration="1.350609003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:08.299336211 +0000 UTC m=+804.970588891" watchObservedRunningTime="2026-11-02 01:38:08.350609003 +0000 UTC m=+805.021861708" Nov 02 01:38:09.194562 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:09.194529 5044 generic.go:356] "Generic (PLEG): container finished" podID="652e50af-8a91-43ff-bb49-bbdf781feefd" containerID="acd61ca1e138ca51879c99ddcee190bd2c8d06866d65c415320e2218eb3130a3" exitCode=0 Nov 02 01:38:09.194992 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:09.194603 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" event={"ID":"652e50af-8a91-43ff-bb49-bbdf781feefd","Type":"ContainerDied","Data":"acd61ca1e138ca51879c99ddcee190bd2c8d06866d65c415320e2218eb3130a3"} Nov 02 01:38:09.195121 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:09.195107 5044 scope.go:122] "RemoveContainer" containerID="acd61ca1e138ca51879c99ddcee190bd2c8d06866d65c415320e2218eb3130a3" Nov 02 01:38:09.196253 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:09.196225 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:38:09.196357 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:09.196301 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:38:09.196357 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:09.196350 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:38:10.218763 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.217329 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" event={"ID":"652e50af-8a91-43ff-bb49-bbdf781feefd","Type":"ContainerStarted","Data":"3ce551f917477de5db7686e019549561678b7a87110d88adf8a607abc00f5320"} Nov 02 01:38:10.551331 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.551299 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p"] Nov 02 01:38:10.554130 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.554114 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" Nov 02 01:38:10.621690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.621569 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nk57n\"" Nov 02 01:38:10.621881 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.621816 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-b71ba5-serving-cert\"" Nov 02 01:38:10.625729 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.621968 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 01:38:10.625729 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.622128 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 01:38:10.625729 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.623109 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 01:38:10.626414 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.626360 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p"] Nov 02 01:38:10.732695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.732377 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnzzn\" (UniqueName: \"kubernetes.io/projected/35a2761d-bbbb-4614-8d91-a2466216e37e-kube-api-access-nnzzn\") pod \"maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p\" (UID: \"35a2761d-bbbb-4614-8d91-a2466216e37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" Nov 02 01:38:10.732695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.732452 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a2761d-bbbb-4614-8d91-a2466216e37e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p\" (UID: \"35a2761d-bbbb-4614-8d91-a2466216e37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" Nov 02 01:38:10.732695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.732543 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a2761d-bbbb-4614-8d91-a2466216e37e-metrics-certs\") pod \"maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p\" (UID: \"35a2761d-bbbb-4614-8d91-a2466216e37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" Nov 02 01:38:10.732695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.732639 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a2761d-bbbb-4614-8d91-a2466216e37e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p\" (UID: \"35a2761d-bbbb-4614-8d91-a2466216e37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" Nov 02 01:38:10.751726 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.751696 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:10.836330 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.836220 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnzzn\" (UniqueName: \"kubernetes.io/projected/35a2761d-bbbb-4614-8d91-a2466216e37e-kube-api-access-nnzzn\") pod \"maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p\" (UID: \"35a2761d-bbbb-4614-8d91-a2466216e37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" Nov 02 01:38:10.836330 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.836307 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a2761d-bbbb-4614-8d91-a2466216e37e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p\" (UID: \"35a2761d-bbbb-4614-8d91-a2466216e37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" Nov 02 01:38:10.836580 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.836404 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a2761d-bbbb-4614-8d91-a2466216e37e-metrics-certs\") pod \"maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p\" (UID: \"35a2761d-bbbb-4614-8d91-a2466216e37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" Nov 02 01:38:10.836580 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.836500 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a2761d-bbbb-4614-8d91-a2466216e37e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p\" (UID: \"35a2761d-bbbb-4614-8d91-a2466216e37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" Nov 02 01:38:10.837380 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.837352 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a2761d-bbbb-4614-8d91-a2466216e37e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p\" (UID: \"35a2761d-bbbb-4614-8d91-a2466216e37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" Nov 02 01:38:10.837775 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:38:10.837757 5044 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-serving-cert: secret "maas-api-e2e-worker-w0-b71ba5-serving-cert" not found Nov 02 01:38:10.837886 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:38:10.837820 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35a2761d-bbbb-4614-8d91-a2466216e37e-maas-api-tls podName:35a2761d-bbbb-4614-8d91-a2466216e37e nodeName:}" failed. No retries permitted until 2026-11-02 01:38:11.337801202 +0000 UTC m=+808.009053879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/35a2761d-bbbb-4614-8d91-a2466216e37e-maas-api-tls") pod "maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" (UID: "35a2761d-bbbb-4614-8d91-a2466216e37e") : secret "maas-api-e2e-worker-w0-b71ba5-serving-cert" not found Nov 02 01:38:10.848252 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.848228 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a2761d-bbbb-4614-8d91-a2466216e37e-metrics-certs\") pod \"maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p\" (UID: \"35a2761d-bbbb-4614-8d91-a2466216e37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" Nov 02 01:38:10.916411 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.916357 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:10.918915 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:10.918888 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnzzn\" (UniqueName: \"kubernetes.io/projected/35a2761d-bbbb-4614-8d91-a2466216e37e-kube-api-access-nnzzn\") pod \"maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p\" (UID: \"35a2761d-bbbb-4614-8d91-a2466216e37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" Nov 02 01:38:11.024277 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:11.024216 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:38:11.025434 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:11.025412 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 02 01:38:11.025597 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:11.025579 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" podUID="652e50af-8a91-43ff-bb49-bbdf781feefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 02 01:38:11.192618 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:11.192510 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:11.342259 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:11.342221 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a2761d-bbbb-4614-8d91-a2466216e37e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p\" (UID: \"35a2761d-bbbb-4614-8d91-a2466216e37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" Nov 02 01:38:11.345261 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:11.345235 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a2761d-bbbb-4614-8d91-a2466216e37e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p\" (UID: \"35a2761d-bbbb-4614-8d91-a2466216e37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" Nov 02 01:38:11.515952 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:11.515911 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" Nov 02 01:38:11.519814 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:11.519788 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:11.717517 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:11.717473 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:11.990591 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:11.990542 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p"] Nov 02 01:38:12.027695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:12.024121 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 02 01:38:12.027695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:12.024182 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" podUID="652e50af-8a91-43ff-bb49-bbdf781feefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 02 01:38:12.140239 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:12.140203 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr" Nov 02 01:38:12.229051 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:12.229019 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" event={"ID":"35a2761d-bbbb-4614-8d91-a2466216e37e","Type":"ContainerStarted","Data":"7d69d2c36a7df0eb127b2541ca7325c1db0870a6bfbca0c9de4a437c413c874d"} Nov 02 01:38:13.023781 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:13.023749 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 02 01:38:13.024372 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:13.024339 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" podUID="652e50af-8a91-43ff-bb49-bbdf781feefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 02 01:38:14.023840 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:14.023805 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 02 01:38:14.024232 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:14.023868 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" podUID="652e50af-8a91-43ff-bb49-bbdf781feefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 02 01:38:14.159704 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:14.157430 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9" Nov 02 01:38:15.023833 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:15.023799 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Nov 02 01:38:15.024016 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:15.023855 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" podUID="652e50af-8a91-43ff-bb49-bbdf781feefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Nov 02 01:38:15.768545 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:15.768506 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:16.028763 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:16.028665 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:38:16.029550 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:16.029526 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:38:16.030366 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:16.030343 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 01:38:16.263752 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:16.263718 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" event={"ID":"35a2761d-bbbb-4614-8d91-a2466216e37e","Type":"ContainerStarted","Data":"d07e2cbe7924444fdec8c52ed7d49152a823df76f402739a1518aa11a9d30e0b"} Nov 02 01:38:16.264007 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:16.263983 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" Nov 02 01:38:16.402078 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:16.401982 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:16.444662 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:16.444610 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" podStartSLOduration=2.302784217 podStartE2EDuration="6.444593295s" totalImagesPullingTime="4.141809078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:16.444149536 +0000 UTC m=+813.115402218" watchObservedRunningTime="2026-11-02 01:38:16.444593295 +0000 UTC m=+813.115845979" Nov 02 01:38:16.580610 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:16.580575 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:16.825015 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:16.824976 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t"] Nov 02 01:38:16.829945 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:16.829917 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t" Nov 02 01:38:16.863391 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:16.863369 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d9zq\" (UniqueName: \"kubernetes.io/projected/6b201831-e055-4763-9400-bb68d825dc3e-kube-api-access-6d9zq\") pod \"payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t\" (UID: \"6b201831-e055-4763-9400-bb68d825dc3e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t" Nov 02 01:38:16.863521 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:16.863425 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b201831-e055-4763-9400-bb68d825dc3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t\" (UID: \"6b201831-e055-4763-9400-bb68d825dc3e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t" Nov 02 01:38:16.883662 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:16.883635 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-b71ba5\"" Nov 02 01:38:16.890391 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:16.890360 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t"] Nov 02 01:38:16.968339 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:16.968295 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b201831-e055-4763-9400-bb68d825dc3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t\" (UID: \"6b201831-e055-4763-9400-bb68d825dc3e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t" Nov 02 01:38:16.968532 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:16.968459 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d9zq\" (UniqueName: \"kubernetes.io/projected/6b201831-e055-4763-9400-bb68d825dc3e-kube-api-access-6d9zq\") pod \"payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t\" (UID: \"6b201831-e055-4763-9400-bb68d825dc3e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t" Nov 02 01:38:16.969474 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:16.969448 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b201831-e055-4763-9400-bb68d825dc3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t\" (UID: \"6b201831-e055-4763-9400-bb68d825dc3e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t" Nov 02 01:38:17.031949 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:17.031912 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d9zq\" (UniqueName: \"kubernetes.io/projected/6b201831-e055-4763-9400-bb68d825dc3e-kube-api-access-6d9zq\") pod \"payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t\" (UID: \"6b201831-e055-4763-9400-bb68d825dc3e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t" Nov 02 01:38:17.143835 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:17.143751 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t" Nov 02 01:38:17.527462 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:17.527438 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t"] Nov 02 01:38:17.528730 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:38:17.528701 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b201831_e055_4763_9400_bb68d825dc3e.slice/crio-0026b2ed5ab73a8bda36a284bb9e33d6f0f64b2cd3e69b51ecbda42898383bb4 WatchSource:0}: Error finding container 0026b2ed5ab73a8bda36a284bb9e33d6f0f64b2cd3e69b51ecbda42898383bb4: Status 404 returned error can't find the container with id 0026b2ed5ab73a8bda36a284bb9e33d6f0f64b2cd3e69b51ecbda42898383bb4 Nov 02 01:38:18.276472 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:18.276434 5044 generic.go:356] "Generic (PLEG): container finished" podID="c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" containerID="cd4da6ec8f1b51383423c155fb319292fa6c7bf32aa9b04bf546d2bccb7655b7" exitCode=0 Nov 02 01:38:18.276955 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:18.276499 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" event={"ID":"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa","Type":"ContainerDied","Data":"cd4da6ec8f1b51383423c155fb319292fa6c7bf32aa9b04bf546d2bccb7655b7"} Nov 02 01:38:18.277020 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:18.276955 5044 scope.go:122] "RemoveContainer" containerID="cd4da6ec8f1b51383423c155fb319292fa6c7bf32aa9b04bf546d2bccb7655b7" Nov 02 01:38:18.278350 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:18.278323 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:38:18.278451 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:18.278354 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t" event={"ID":"6b201831-e055-4763-9400-bb68d825dc3e","Type":"ContainerStarted","Data":"0f5eb9ffef9a671465a144df12943bf6c4164d979907217d195101583dbfbf83"} Nov 02 01:38:18.278451 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:18.278398 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t" event={"ID":"6b201831-e055-4763-9400-bb68d825dc3e","Type":"ContainerStarted","Data":"0026b2ed5ab73a8bda36a284bb9e33d6f0f64b2cd3e69b51ecbda42898383bb4"} Nov 02 01:38:18.278451 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:18.278408 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:38:18.278451 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:18.278442 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t" Nov 02 01:38:18.278656 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:18.278457 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:38:18.516725 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:18.516643 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t" podStartSLOduration=2.51662915 podStartE2EDuration="2.51662915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:18.516178226 +0000 UTC m=+815.187430908" watchObservedRunningTime="2026-11-02 01:38:18.51662915 +0000 UTC m=+815.187881895" Nov 02 01:38:19.198302 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:19.198262 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2" Nov 02 01:38:19.296812 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:19.296778 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" event={"ID":"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa","Type":"ContainerStarted","Data":"e0e414d7fd49c210ddf30c115581a93bd5c5889d055ae5bd7bf0c447b01ee479"} Nov 02 01:38:19.696624 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:19.696592 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:38:19.697797 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:19.697773 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 02 01:38:19.697921 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:19.697816 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" podUID="c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 02 01:38:20.702690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:20.698083 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 02 01:38:20.702690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:20.698141 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" podUID="c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 02 01:38:20.727064 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:20.727012 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:20.903986 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:20.903951 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:21.052851 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:21.052805 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:21.697240 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:21.697192 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 02 01:38:21.697447 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:21.697268 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" podUID="c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 02 01:38:21.993702 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:21.993607 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:38:22.287170 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:22.287136 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" Nov 02 01:38:22.697222 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:22.697130 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 02 01:38:22.697222 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:22.697185 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" podUID="c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 02 01:38:23.697268 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:23.697233 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 02 01:38:23.697697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:23.697288 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" podUID="c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 02 01:38:24.192982 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:38:24.192944 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:38:24.697055 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:24.697019 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 02 01:38:24.697236 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:24.697104 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" podUID="c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 02 01:38:25.501881 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:25.501851 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:25.701769 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:25.701742 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:38:25.702908 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:25.702886 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 01:38:25.865254 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:25.865199 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f"] Nov 02 01:38:25.872413 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:25.872383 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" Nov 02 01:38:25.926009 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:25.925982 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-5888dc-serving-cert\"" Nov 02 01:38:25.932945 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:25.932915 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f"] Nov 02 01:38:25.964257 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:25.964217 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtpkb\" (UniqueName: \"kubernetes.io/projected/7812bf12-6286-4de7-88b4-b6d509c9482c-kube-api-access-qtpkb\") pod \"maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f\" (UID: \"7812bf12-6286-4de7-88b4-b6d509c9482c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" Nov 02 01:38:25.964401 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:25.964285 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7812bf12-6286-4de7-88b4-b6d509c9482c-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f\" (UID: \"7812bf12-6286-4de7-88b4-b6d509c9482c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" Nov 02 01:38:25.964401 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:25.964380 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7812bf12-6286-4de7-88b4-b6d509c9482c-metrics-certs\") pod \"maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f\" (UID: \"7812bf12-6286-4de7-88b4-b6d509c9482c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" Nov 02 01:38:25.964473 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:25.964424 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7812bf12-6286-4de7-88b4-b6d509c9482c-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f\" (UID: \"7812bf12-6286-4de7-88b4-b6d509c9482c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" Nov 02 01:38:26.065991 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:26.065949 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7812bf12-6286-4de7-88b4-b6d509c9482c-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f\" (UID: \"7812bf12-6286-4de7-88b4-b6d509c9482c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" Nov 02 01:38:26.066153 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:26.066014 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7812bf12-6286-4de7-88b4-b6d509c9482c-metrics-certs\") pod \"maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f\" (UID: \"7812bf12-6286-4de7-88b4-b6d509c9482c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" Nov 02 01:38:26.066153 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:26.066068 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7812bf12-6286-4de7-88b4-b6d509c9482c-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f\" (UID: \"7812bf12-6286-4de7-88b4-b6d509c9482c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" Nov 02 01:38:26.066153 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:38:26.066103 5044 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-serving-cert: secret "maas-api-e2e-worker-w6-5888dc-serving-cert" not found Nov 02 01:38:26.066303 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:38:26.066161 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7812bf12-6286-4de7-88b4-b6d509c9482c-maas-api-tls podName:7812bf12-6286-4de7-88b4-b6d509c9482c nodeName:}" failed. No retries permitted until 2026-11-02 01:38:26.566143783 +0000 UTC m=+823.237396465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7812bf12-6286-4de7-88b4-b6d509c9482c-maas-api-tls") pod "maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" (UID: "7812bf12-6286-4de7-88b4-b6d509c9482c") : secret "maas-api-e2e-worker-w6-5888dc-serving-cert" not found Nov 02 01:38:26.066303 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:26.066244 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtpkb\" (UniqueName: \"kubernetes.io/projected/7812bf12-6286-4de7-88b4-b6d509c9482c-kube-api-access-qtpkb\") pod \"maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f\" (UID: \"7812bf12-6286-4de7-88b4-b6d509c9482c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" Nov 02 01:38:26.066982 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:26.066961 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7812bf12-6286-4de7-88b4-b6d509c9482c-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f\" (UID: \"7812bf12-6286-4de7-88b4-b6d509c9482c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" Nov 02 01:38:26.068538 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:26.068514 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7812bf12-6286-4de7-88b4-b6d509c9482c-metrics-certs\") pod \"maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f\" (UID: \"7812bf12-6286-4de7-88b4-b6d509c9482c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" Nov 02 01:38:26.071820 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:26.071801 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:26.137301 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:26.137227 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtpkb\" (UniqueName: \"kubernetes.io/projected/7812bf12-6286-4de7-88b4-b6d509c9482c-kube-api-access-qtpkb\") pod \"maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f\" (UID: \"7812bf12-6286-4de7-88b4-b6d509c9482c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" Nov 02 01:38:26.225391 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:26.225355 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:26.372634 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:26.372598 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:26.571408 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:26.571372 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7812bf12-6286-4de7-88b4-b6d509c9482c-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f\" (UID: \"7812bf12-6286-4de7-88b4-b6d509c9482c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" Nov 02 01:38:26.573868 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:26.573844 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7812bf12-6286-4de7-88b4-b6d509c9482c-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f\" (UID: \"7812bf12-6286-4de7-88b4-b6d509c9482c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" Nov 02 01:38:26.705470 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:26.705435 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:26.791148 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:26.785665 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" Nov 02 01:38:26.857814 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:26.857781 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:27.178280 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:27.178244 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f"] Nov 02 01:38:27.180386 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:38:27.180357 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7812bf12_6286_4de7_88b4_b6d509c9482c.slice/crio-a1940279cec282b250b0d32778558361a6aebbf8d8b3c6232a6cad1a1bb50ad7 WatchSource:0}: Error finding container a1940279cec282b250b0d32778558361a6aebbf8d8b3c6232a6cad1a1bb50ad7: Status 404 returned error can't find the container with id a1940279cec282b250b0d32778558361a6aebbf8d8b3c6232a6cad1a1bb50ad7 Nov 02 01:38:27.347125 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:27.347094 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" event={"ID":"7812bf12-6286-4de7-88b4-b6d509c9482c","Type":"ContainerStarted","Data":"a1940279cec282b250b0d32778558361a6aebbf8d8b3c6232a6cad1a1bb50ad7"} Nov 02 01:38:28.355244 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:28.355152 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" event={"ID":"7812bf12-6286-4de7-88b4-b6d509c9482c","Type":"ContainerStarted","Data":"dcb5f8918141f533cf4b766953633bfa83292f32a9d8afdb75562b5d6478afd6"} Nov 02 01:38:28.355244 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:28.355210 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" Nov 02 01:38:28.474079 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:28.474031 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" podStartSLOduration=2.6402216640000002 podStartE2EDuration="3.474018037s" totalImagesPullingTime="833.796373ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:28.47393074 +0000 UTC m=+825.145183421" watchObservedRunningTime="2026-11-02 01:38:28.474018037 +0000 UTC m=+825.145270717" Nov 02 01:38:29.300350 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:29.300323 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t" Nov 02 01:38:31.399679 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:31.399633 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:31.734786 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:31.734710 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:31.912327 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:31.912273 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:32.147988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.147958 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d"] Nov 02 01:38:32.151868 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.151848 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d" Nov 02 01:38:32.206946 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.206918 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-5888dc\"" Nov 02 01:38:32.210010 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.209986 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d"] Nov 02 01:38:32.234621 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.234596 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hxcs\" (UniqueName: \"kubernetes.io/projected/0f4ffc26-b174-4586-aa5d-11b08755848a-kube-api-access-6hxcs\") pod \"payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d\" (UID: \"0f4ffc26-b174-4586-aa5d-11b08755848a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d" Nov 02 01:38:32.234784 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.234666 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4ffc26-b174-4586-aa5d-11b08755848a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d\" (UID: \"0f4ffc26-b174-4586-aa5d-11b08755848a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d" Nov 02 01:38:32.313300 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.313264 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l"] Nov 02 01:38:32.322885 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.322853 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l" Nov 02 01:38:32.336072 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.336033 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hxcs\" (UniqueName: \"kubernetes.io/projected/0f4ffc26-b174-4586-aa5d-11b08755848a-kube-api-access-6hxcs\") pod \"payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d\" (UID: \"0f4ffc26-b174-4586-aa5d-11b08755848a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d" Nov 02 01:38:32.336227 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.336147 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4ffc26-b174-4586-aa5d-11b08755848a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d\" (UID: \"0f4ffc26-b174-4586-aa5d-11b08755848a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d" Nov 02 01:38:32.337088 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.337062 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4ffc26-b174-4586-aa5d-11b08755848a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d\" (UID: \"0f4ffc26-b174-4586-aa5d-11b08755848a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d" Nov 02 01:38:32.364916 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.364887 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l"] Nov 02 01:38:32.376973 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.376950 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-5888dc-dockercfg-jqwch\"" Nov 02 01:38:32.399401 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.399325 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hxcs\" (UniqueName: \"kubernetes.io/projected/0f4ffc26-b174-4586-aa5d-11b08755848a-kube-api-access-6hxcs\") pod \"payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d\" (UID: \"0f4ffc26-b174-4586-aa5d-11b08755848a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d" Nov 02 01:38:32.437344 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.437312 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnb9l\" (UniqueName: \"kubernetes.io/projected/d7d1b5a4-50ae-46d5-83f9-252cb2e8734c-kube-api-access-cnb9l\") pod \"payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l\" (UID: \"d7d1b5a4-50ae-46d5-83f9-252cb2e8734c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l" Nov 02 01:38:32.437344 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.437344 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d1b5a4-50ae-46d5-83f9-252cb2e8734c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l\" (UID: \"d7d1b5a4-50ae-46d5-83f9-252cb2e8734c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l" Nov 02 01:38:32.464837 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.464794 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d" Nov 02 01:38:32.538279 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.538244 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnb9l\" (UniqueName: \"kubernetes.io/projected/d7d1b5a4-50ae-46d5-83f9-252cb2e8734c-kube-api-access-cnb9l\") pod \"payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l\" (UID: \"d7d1b5a4-50ae-46d5-83f9-252cb2e8734c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l" Nov 02 01:38:32.538404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.538289 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d1b5a4-50ae-46d5-83f9-252cb2e8734c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l\" (UID: \"d7d1b5a4-50ae-46d5-83f9-252cb2e8734c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l" Nov 02 01:38:32.539075 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.539053 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d1b5a4-50ae-46d5-83f9-252cb2e8734c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l\" (UID: \"d7d1b5a4-50ae-46d5-83f9-252cb2e8734c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l" Nov 02 01:38:32.599873 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.599843 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnb9l\" (UniqueName: \"kubernetes.io/projected/d7d1b5a4-50ae-46d5-83f9-252cb2e8734c-kube-api-access-cnb9l\") pod \"payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l\" (UID: \"d7d1b5a4-50ae-46d5-83f9-252cb2e8734c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l" Nov 02 01:38:32.639317 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.639286 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l" Nov 02 01:38:32.839297 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:32.839247 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d"] Nov 02 01:38:32.842228 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:38:32.842196 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4ffc26_b174_4586_aa5d_11b08755848a.slice/crio-569bd9f6ebce564993cc4cc95c10effea5af42cd22e73496bddcd69d698859c5 WatchSource:0}: Error finding container 569bd9f6ebce564993cc4cc95c10effea5af42cd22e73496bddcd69d698859c5: Status 404 returned error can't find the container with id 569bd9f6ebce564993cc4cc95c10effea5af42cd22e73496bddcd69d698859c5 Nov 02 01:38:33.021457 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:33.021433 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l"] Nov 02 01:38:33.023030 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:38:33.023001 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d1b5a4_50ae_46d5_83f9_252cb2e8734c.slice/crio-7390f611dc33b13b17c2254a8f24bfa3a8941b39fcdaf35459b7c8dd0730b870 WatchSource:0}: Error finding container 7390f611dc33b13b17c2254a8f24bfa3a8941b39fcdaf35459b7c8dd0730b870: Status 404 returned error can't find the container with id 7390f611dc33b13b17c2254a8f24bfa3a8941b39fcdaf35459b7c8dd0730b870 Nov 02 01:38:33.399377 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:33.399339 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d" event={"ID":"0f4ffc26-b174-4586-aa5d-11b08755848a","Type":"ContainerStarted","Data":"fb06780a486173a602bc7d8011c3e6ae3b72cfa268f333214be6450bde7e8f12"} Nov 02 01:38:33.399539 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:33.399532 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d" Nov 02 01:38:33.399619 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:33.399560 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d" event={"ID":"0f4ffc26-b174-4586-aa5d-11b08755848a","Type":"ContainerStarted","Data":"569bd9f6ebce564993cc4cc95c10effea5af42cd22e73496bddcd69d698859c5"} Nov 02 01:38:33.401022 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:33.400988 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l" event={"ID":"d7d1b5a4-50ae-46d5-83f9-252cb2e8734c","Type":"ContainerStarted","Data":"26e6bb3a3bfda4dc8a52a5c0eb2d8290eb216a96ddfe919f6ca282d7f9a37474"} Nov 02 01:38:33.401157 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:33.401032 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l" event={"ID":"d7d1b5a4-50ae-46d5-83f9-252cb2e8734c","Type":"ContainerStarted","Data":"7390f611dc33b13b17c2254a8f24bfa3a8941b39fcdaf35459b7c8dd0730b870"} Nov 02 01:38:33.401157 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:33.401063 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l" Nov 02 01:38:33.517914 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:33.517862 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d" podStartSLOduration=1.517848704 podStartE2EDuration="1.517848704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:33.516763339 +0000 UTC m=+830.188016021" watchObservedRunningTime="2026-11-02 01:38:33.517848704 +0000 UTC m=+830.189101384" Nov 02 01:38:33.638199 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:33.638148 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l" podStartSLOduration=1.638134464 podStartE2EDuration="1.638134464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:33.636490078 +0000 UTC m=+830.307742759" watchObservedRunningTime="2026-11-02 01:38:33.638134464 +0000 UTC m=+830.309387170" Nov 02 01:38:34.366332 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:34.366302 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" Nov 02 01:38:36.777283 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:36.777247 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:37.573110 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:37.573078 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:41.118126 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:41.118079 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:41.718535 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:41.718495 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:41.886304 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:41.886258 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:42.038187 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:42.038159 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:44.413930 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:44.413901 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d" Nov 02 01:38:44.414327 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:44.413965 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l" Nov 02 01:38:45.121232 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.121201 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7"] Nov 02 01:38:45.126150 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.126132 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:45.196192 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.196156 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggfss\" (UniqueName: \"kubernetes.io/projected/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-kube-api-access-ggfss\") pod \"simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:45.196371 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.196225 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-model-cache\") pod \"simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:45.196371 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.196247 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-tmp-dir\") pod \"simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:45.196371 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.196268 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-dshm\") pod \"simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:45.196371 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.196290 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-tls-certs\") pod \"simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:45.196371 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.196339 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-home\") pod \"simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:45.216385 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.216346 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:45.242867 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.242839 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e06ea2\"/\"default-dockercfg-mqqzb\"" Nov 02 01:38:45.242867 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.242856 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e06ea2\"/\"openshift-service-ca.crt\"" Nov 02 01:38:45.243077 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.243027 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e06ea2\"/\"kube-root-ca.crt\"" Nov 02 01:38:45.243186 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.243171 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e06ea2\"/\"simulator-w5-e06ea2-kserve-self-signed-certs\"" Nov 02 01:38:45.245689 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.245653 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7"] Nov 02 01:38:45.299742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.299359 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-home\") pod \"simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:45.299742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.299451 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggfss\" (UniqueName: \"kubernetes.io/projected/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-kube-api-access-ggfss\") pod \"simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:45.299742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.299538 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-model-cache\") pod \"simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:45.299742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.299572 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-tmp-dir\") pod \"simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:45.299742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.299604 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-dshm\") pod \"simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:45.299742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.299638 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-tls-certs\") pod \"simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:45.300865 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.300809 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-model-cache\") pod \"simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:45.302406 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.302351 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-tmp-dir\") pod \"simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:45.303365 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.302897 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-home\") pod \"simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:45.305072 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.305048 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-dshm\") pod \"simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:45.305866 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.305845 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-tls-certs\") pod \"simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:45.394382 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.394293 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggfss\" (UniqueName: \"kubernetes.io/projected/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-kube-api-access-ggfss\") pod \"simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:45.437984 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.437954 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:45.851646 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:45.851613 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7"] Nov 02 01:38:45.852961 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:38:45.852936 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c1ae51e_d552_4133_a0d5_5628c15cf7c7.slice/crio-de93d781a5f01f074259b569eaf0c92e98dba42e4eea13b8f18244377ad51113 WatchSource:0}: Error finding container de93d781a5f01f074259b569eaf0c92e98dba42e4eea13b8f18244377ad51113: Status 404 returned error can't find the container with id de93d781a5f01f074259b569eaf0c92e98dba42e4eea13b8f18244377ad51113 Nov 02 01:38:46.458431 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:46.458402 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:46.507600 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:46.507565 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" event={"ID":"8c1ae51e-d552-4133-a0d5-5628c15cf7c7","Type":"ContainerStarted","Data":"6fb5591c86f2d36ec6cac463c67227b96f429e5ed6148dbc96f52a055b70b248"} Nov 02 01:38:46.507600 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:46.507607 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" event={"ID":"8c1ae51e-d552-4133-a0d5-5628c15cf7c7","Type":"ContainerStarted","Data":"de93d781a5f01f074259b569eaf0c92e98dba42e4eea13b8f18244377ad51113"} Nov 02 01:38:46.640131 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:46.632252 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podStartSLOduration=1.632221168 podStartE2EDuration="1.632221168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:38:46.630480501 +0000 UTC m=+843.301733184" watchObservedRunningTime="2026-11-02 01:38:46.632221168 +0000 UTC m=+843.303473850" Nov 02 01:38:47.053179 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:47.053104 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:47.200112 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:47.200078 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:47.350032 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:47.349958 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:47.438480 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:47.438448 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:47.497216 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:47.497184 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:50.462458 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:50.462424 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:50.531424 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:50.531391 5044 generic.go:356] "Generic (PLEG): container finished" podID="b7d46753-0b7a-4c8c-9870-31cedadbe4c5" containerID="b490bbb8a5b53dfe58e293e00dcb367abc4b391097731dc7baa85a5acc11329b" exitCode=0 Nov 02 01:38:50.531617 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:50.531463 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" event={"ID":"b7d46753-0b7a-4c8c-9870-31cedadbe4c5","Type":"ContainerDied","Data":"b490bbb8a5b53dfe58e293e00dcb367abc4b391097731dc7baa85a5acc11329b"} Nov 02 01:38:50.531873 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:50.531860 5044 scope.go:122] "RemoveContainer" containerID="b490bbb8a5b53dfe58e293e00dcb367abc4b391097731dc7baa85a5acc11329b" Nov 02 01:38:50.533181 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:50.533158 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:38:50.533248 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:50.533226 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:38:50.533283 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:50.533260 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:38:51.078782 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:51.078750 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:51.288571 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:51.288451 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:51.435083 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:51.434984 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:51.538484 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:51.538453 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" event={"ID":"b7d46753-0b7a-4c8c-9870-31cedadbe4c5","Type":"ContainerStarted","Data":"215d1088a6457e2da21f285503ff09dca44db2d4b7eda1d4c13e2afff4a6915b"} Nov 02 01:38:51.635385 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:51.635350 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:38:51.636697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:51.636660 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 02 01:38:51.636800 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:51.636726 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" podUID="b7d46753-0b7a-4c8c-9870-31cedadbe4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 02 01:38:52.635641 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:52.635602 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 02 01:38:52.636242 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:52.635692 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" podUID="b7d46753-0b7a-4c8c-9870-31cedadbe4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 02 01:38:52.786443 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:52.786351 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:52.999526 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:52.999432 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:38:53.635463 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:53.635430 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 02 01:38:53.635637 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:53.635488 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" podUID="b7d46753-0b7a-4c8c-9870-31cedadbe4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 02 01:38:54.193395 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:38:54.193364 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:38:54.635205 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:54.635170 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 02 01:38:54.635382 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:54.635233 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" podUID="b7d46753-0b7a-4c8c-9870-31cedadbe4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 02 01:38:55.438917 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:55.438883 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:55.452265 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:55.452241 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:55.595834 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:55.595805 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:38:55.635466 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:55.635436 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 02 01:38:55.635638 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:55.635487 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" podUID="b7d46753-0b7a-4c8c-9870-31cedadbe4c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 02 01:38:56.681418 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:56.681381 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:38:56.683165 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:56.683139 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:38:56.858160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:56.858131 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:57.468063 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:57.468026 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:57.621047 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:57.621012 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:57.766595 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:57.766562 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:38:57.924520 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:38:57.924488 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:01.127774 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:01.127726 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:01.783159 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:01.783129 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:01.933076 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:01.933045 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:02.079175 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:02.079088 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:02.996620 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:02.996593 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:07.871764 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:07.871711 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:09.111579 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:09.111545 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:11.557286 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:11.557243 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:12.230102 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:12.230067 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:12.385919 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:12.385878 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:12.539041 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:12.539007 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:12.893647 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:12.893569 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:15.208065 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.208035 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm"] Nov 02 01:39:15.213075 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.213055 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:15.318372 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.318340 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f276c3\"/\"kube-root-ca.crt\"" Nov 02 01:39:15.318559 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.318418 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f276c3\"/\"openshift-service-ca.crt\"" Nov 02 01:39:15.318559 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.318481 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f276c3\"/\"default-dockercfg-sphkv\"" Nov 02 01:39:15.318559 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.318487 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f276c3\"/\"simulator-w1-f276c3-kserve-self-signed-certs\"" Nov 02 01:39:15.320608 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.320585 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm"] Nov 02 01:39:15.349168 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.349118 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-tls-certs\") pod \"simulator-w1-f276c3-kserve-574fb98789-7m7bm\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:15.349168 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.349170 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-model-cache\") pod \"simulator-w1-f276c3-kserve-574fb98789-7m7bm\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:15.349521 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.349201 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-tmp-dir\") pod \"simulator-w1-f276c3-kserve-574fb98789-7m7bm\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:15.349521 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.349280 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-home\") pod \"simulator-w1-f276c3-kserve-574fb98789-7m7bm\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:15.349521 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.349315 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s2zd\" (UniqueName: \"kubernetes.io/projected/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-kube-api-access-8s2zd\") pod \"simulator-w1-f276c3-kserve-574fb98789-7m7bm\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:15.349521 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.349459 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-dshm\") pod \"simulator-w1-f276c3-kserve-574fb98789-7m7bm\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:15.402057 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.402021 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:15.450071 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.450041 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-dshm\") pod \"simulator-w1-f276c3-kserve-574fb98789-7m7bm\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:15.450230 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.450091 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-tls-certs\") pod \"simulator-w1-f276c3-kserve-574fb98789-7m7bm\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:15.450230 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.450110 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-model-cache\") pod \"simulator-w1-f276c3-kserve-574fb98789-7m7bm\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:15.450230 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.450127 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-tmp-dir\") pod \"simulator-w1-f276c3-kserve-574fb98789-7m7bm\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:15.450230 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.450192 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-home\") pod \"simulator-w1-f276c3-kserve-574fb98789-7m7bm\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:15.450230 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.450224 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s2zd\" (UniqueName: \"kubernetes.io/projected/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-kube-api-access-8s2zd\") pod \"simulator-w1-f276c3-kserve-574fb98789-7m7bm\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:15.450628 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.450599 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-model-cache\") pod \"simulator-w1-f276c3-kserve-574fb98789-7m7bm\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:15.450802 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.450775 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-home\") pod \"simulator-w1-f276c3-kserve-574fb98789-7m7bm\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:15.451012 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.450991 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-tmp-dir\") pod \"simulator-w1-f276c3-kserve-574fb98789-7m7bm\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:15.452509 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.452486 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-dshm\") pod \"simulator-w1-f276c3-kserve-574fb98789-7m7bm\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:15.452851 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.452834 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-tls-certs\") pod \"simulator-w1-f276c3-kserve-574fb98789-7m7bm\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:15.512243 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.512212 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s2zd\" (UniqueName: \"kubernetes.io/projected/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-kube-api-access-8s2zd\") pod \"simulator-w1-f276c3-kserve-574fb98789-7m7bm\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:15.524784 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.524759 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:15.912061 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:15.912033 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm"] Nov 02 01:39:15.913022 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:39:15.912995 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a1661ef_7b1a_48fe_9514_e1ffe7ba7a94.slice/crio-0551229e765859f8cfd7af50529c4b689ad7d10b76c7f5e2301e8890470c2902 WatchSource:0}: Error finding container 0551229e765859f8cfd7af50529c4b689ad7d10b76c7f5e2301e8890470c2902: Status 404 returned error can't find the container with id 0551229e765859f8cfd7af50529c4b689ad7d10b76c7f5e2301e8890470c2902 Nov 02 01:39:16.706710 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:16.706660 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" event={"ID":"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94","Type":"ContainerStarted","Data":"c3c8402ba52cfd2bc608c16c6c126655230f62d55aaf53cc366a57b3fec471c2"} Nov 02 01:39:16.706710 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:16.706713 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" event={"ID":"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94","Type":"ContainerStarted","Data":"0551229e765859f8cfd7af50529c4b689ad7d10b76c7f5e2301e8890470c2902"} Nov 02 01:39:16.826894 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:16.826841 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podStartSLOduration=1.826827315 podStartE2EDuration="1.826827315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:16.824978895 +0000 UTC m=+873.496231576" watchObservedRunningTime="2026-11-02 01:39:16.826827315 +0000 UTC m=+873.498079996" Nov 02 01:39:16.946400 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:16.946363 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:17.525898 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:17.525870 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:17.552401 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:17.552339 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:17.701076 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:17.701046 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:17.855417 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:17.855323 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:18.001314 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:18.001284 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:22.588900 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:22.588868 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:22.740974 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:22.740948 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:24.172327 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:39:24.172290 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:39:25.525351 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:25.525320 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:25.537664 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:25.537643 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:25.793733 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:25.793654 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:39:27.474827 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:27.474799 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:32.197551 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.197508 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:32.317239 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.317201 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5"] Nov 02 01:39:32.321350 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.321325 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:32.376821 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.376786 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b71ba5\"/\"openshift-service-ca.crt\"" Nov 02 01:39:32.417118 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.417079 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:32.427351 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.427325 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-tmp-dir\") pod \"simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:32.427483 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.427434 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-dshm\") pod \"simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:32.427483 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.427463 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-model-cache\") pod \"simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:32.427598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.427496 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-home\") pod \"simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:32.427598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.427570 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwxwt\" (UniqueName: \"kubernetes.io/projected/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-kube-api-access-dwxwt\") pod \"simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:32.427724 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.427651 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b71ba5\"/\"default-dockercfg-gxmsh\"" Nov 02 01:39:32.427724 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.427649 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b71ba5\"/\"kube-root-ca.crt\"" Nov 02 01:39:32.427850 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.427786 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b71ba5\"/\"simulator-w0-b71ba5-kserve-self-signed-certs\"" Nov 02 01:39:32.427850 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.427786 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-tls-certs\") pod \"simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:32.430205 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.430185 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5"] Nov 02 01:39:32.529055 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.529024 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-tls-certs\") pod \"simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:32.529232 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.529060 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-tmp-dir\") pod \"simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:32.529232 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.529118 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-dshm\") pod \"simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:32.529232 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.529134 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-model-cache\") pod \"simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:32.529232 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.529150 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-home\") pod \"simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:32.529431 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.529279 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwxwt\" (UniqueName: \"kubernetes.io/projected/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-kube-api-access-dwxwt\") pod \"simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:32.529594 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.529572 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-model-cache\") pod \"simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:32.529696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.529594 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-tmp-dir\") pod \"simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:32.529696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.529654 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-home\") pod \"simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:32.531443 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.531420 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-dshm\") pod \"simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:32.531610 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.531592 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-tls-certs\") pod \"simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:32.591406 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.591378 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwxwt\" (UniqueName: \"kubernetes.io/projected/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-kube-api-access-dwxwt\") pod \"simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:32.634622 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:32.634599 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:33.011255 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:33.011222 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5"] Nov 02 01:39:33.012002 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:39:33.011973 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4e6f149_8398_4eb5_8fcc_f658c6e3300c.slice/crio-bdd73960cf179e7aa2119bbdb7277bd33c507fc32ddf87efadfb1b06d693a8e9 WatchSource:0}: Error finding container bdd73960cf179e7aa2119bbdb7277bd33c507fc32ddf87efadfb1b06d693a8e9: Status 404 returned error can't find the container with id bdd73960cf179e7aa2119bbdb7277bd33c507fc32ddf87efadfb1b06d693a8e9 Nov 02 01:39:33.831105 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:33.831071 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" event={"ID":"a4e6f149-8398-4eb5-8fcc-f658c6e3300c","Type":"ContainerStarted","Data":"3c1b871342495601c96c323ca4ecaa78b0341a4d166261e399a9b3263f50689b"} Nov 02 01:39:33.831105 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:33.831109 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" event={"ID":"a4e6f149-8398-4eb5-8fcc-f658c6e3300c","Type":"ContainerStarted","Data":"bdd73960cf179e7aa2119bbdb7277bd33c507fc32ddf87efadfb1b06d693a8e9"} Nov 02 01:39:33.951620 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:33.951558 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podStartSLOduration=1.951538051 podStartE2EDuration="1.951538051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:33.949012797 +0000 UTC m=+890.620265482" watchObservedRunningTime="2026-11-02 01:39:33.951538051 +0000 UTC m=+890.622790734" Nov 02 01:39:34.635167 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:34.635129 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:36.966861 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:36.966825 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:37.919483 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:37.919451 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:41.750957 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:41.750922 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:42.635726 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:42.635697 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:42.648213 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:42.648191 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:42.892954 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:42.892872 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:39:43.525122 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:43.525086 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 01:39:43.525849 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:43.525827 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 01:39:45.171395 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.171353 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv"] Nov 02 01:39:45.175729 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.175709 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:45.232077 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.232048 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv"] Nov 02 01:39:45.256962 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.256931 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:45.269333 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.269309 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-dshm\") pod \"premium-w5-e06ea2-kserve-5bb5554468-xk4wv\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:45.269473 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.269341 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbv6g\" (UniqueName: \"kubernetes.io/projected/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-kube-api-access-vbv6g\") pod \"premium-w5-e06ea2-kserve-5bb5554468-xk4wv\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:45.269473 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.269370 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-tmp-dir\") pod \"premium-w5-e06ea2-kserve-5bb5554468-xk4wv\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:45.269473 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.269437 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-tls-certs\") pod \"premium-w5-e06ea2-kserve-5bb5554468-xk4wv\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:45.269605 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.269580 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-model-cache\") pod \"premium-w5-e06ea2-kserve-5bb5554468-xk4wv\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:45.269646 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.269610 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-home\") pod \"premium-w5-e06ea2-kserve-5bb5554468-xk4wv\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:45.279491 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.279466 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e06ea2\"/\"premium-w5-e06ea2-kserve-self-signed-certs\"" Nov 02 01:39:45.370268 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.370230 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-tmp-dir\") pod \"premium-w5-e06ea2-kserve-5bb5554468-xk4wv\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:45.370549 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.370286 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-tls-certs\") pod \"premium-w5-e06ea2-kserve-5bb5554468-xk4wv\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:45.370549 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.370415 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-model-cache\") pod \"premium-w5-e06ea2-kserve-5bb5554468-xk4wv\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:45.370549 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.370456 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-home\") pod \"premium-w5-e06ea2-kserve-5bb5554468-xk4wv\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:45.370549 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.370533 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-dshm\") pod \"premium-w5-e06ea2-kserve-5bb5554468-xk4wv\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:45.370809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.370560 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbv6g\" (UniqueName: \"kubernetes.io/projected/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-kube-api-access-vbv6g\") pod \"premium-w5-e06ea2-kserve-5bb5554468-xk4wv\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:45.370809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.370662 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-tmp-dir\") pod \"premium-w5-e06ea2-kserve-5bb5554468-xk4wv\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:45.370958 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.370933 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-home\") pod \"premium-w5-e06ea2-kserve-5bb5554468-xk4wv\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:45.371128 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.371079 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-model-cache\") pod \"premium-w5-e06ea2-kserve-5bb5554468-xk4wv\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:45.373039 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.373017 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-dshm\") pod \"premium-w5-e06ea2-kserve-5bb5554468-xk4wv\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:45.374480 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.374460 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-tls-certs\") pod \"premium-w5-e06ea2-kserve-5bb5554468-xk4wv\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:45.432300 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.432234 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbv6g\" (UniqueName: \"kubernetes.io/projected/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-kube-api-access-vbv6g\") pod \"premium-w5-e06ea2-kserve-5bb5554468-xk4wv\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:45.488272 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.488232 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:45.892765 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:45.892704 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv"] Nov 02 01:39:46.572076 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:46.572045 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:46.908094 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:46.908010 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" event={"ID":"3d7bd8f0-4f20-417d-8bdb-c6daecef3395","Type":"ContainerStarted","Data":"7ac72594e91fb6d0bb61a26c231f86b11ae59b23e866a700ef3243ffb0809efa"} Nov 02 01:39:46.908094 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:46.908050 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" event={"ID":"3d7bd8f0-4f20-417d-8bdb-c6daecef3395","Type":"ContainerStarted","Data":"2a79c4c6bb1f5a07c2d10e878a3e2a030997d7cd1ee6e4870cb5530c3e15a020"} Nov 02 01:39:47.026812 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:47.026766 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podStartSLOduration=2.02675389 podStartE2EDuration="2.02675389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:39:47.026455179 +0000 UTC m=+903.697707862" watchObservedRunningTime="2026-11-02 01:39:47.02675389 +0000 UTC m=+903.698006570" Nov 02 01:39:47.488893 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:47.488866 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:51.373013 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:51.372978 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:54.172161 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:39:54.172118 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:39:55.101642 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:55.101608 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:55.500779 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:55.500694 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:55.532484 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:55.532447 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:55.979867 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:55.979835 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:39:56.180768 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:56.180740 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:39:58.735477 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:39:58.735444 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:40:00.975778 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:00.975748 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:40:05.778562 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:05.778521 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:40:10.481326 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:10.481290 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:40:15.329970 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:15.329933 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:40:19.327572 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:19.327537 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:40:20.089251 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:20.089212 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:40:24.173531 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:40:24.173435 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:40:24.806861 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:24.806826 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:40:29.578645 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:29.578617 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:40:34.384799 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:34.384766 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:40:38.251525 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:38.251492 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:40:39.168027 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:39.167997 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:40:40.765334 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.765303 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26"] Nov 02 01:40:40.776927 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.776907 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:40.827004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.826973 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26"] Nov 02 01:40:40.852370 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.852345 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:40:40.858782 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.858759 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-model-cache\") pod \"premium-w1-f276c3-kserve-75c9f975b8-z2g26\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:40.858897 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.858808 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d728t\" (UniqueName: \"kubernetes.io/projected/4e539f99-221e-4050-87f3-a7bd2d1fbb71-kube-api-access-d728t\") pod \"premium-w1-f276c3-kserve-75c9f975b8-z2g26\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:40.858897 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.858879 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-dshm\") pod \"premium-w1-f276c3-kserve-75c9f975b8-z2g26\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:40.859006 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.858982 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-home\") pod \"premium-w1-f276c3-kserve-75c9f975b8-z2g26\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:40.859058 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.859044 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-tmp-dir\") pod \"premium-w1-f276c3-kserve-75c9f975b8-z2g26\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:40.859095 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.859067 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e539f99-221e-4050-87f3-a7bd2d1fbb71-tls-certs\") pod \"premium-w1-f276c3-kserve-75c9f975b8-z2g26\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:40.874703 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.874661 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f276c3\"/\"premium-w1-f276c3-kserve-self-signed-certs\"" Nov 02 01:40:40.960164 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.960131 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-tmp-dir\") pod \"premium-w1-f276c3-kserve-75c9f975b8-z2g26\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:40.960164 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.960169 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e539f99-221e-4050-87f3-a7bd2d1fbb71-tls-certs\") pod \"premium-w1-f276c3-kserve-75c9f975b8-z2g26\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:40.960389 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.960253 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-model-cache\") pod \"premium-w1-f276c3-kserve-75c9f975b8-z2g26\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:40.960389 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.960296 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d728t\" (UniqueName: \"kubernetes.io/projected/4e539f99-221e-4050-87f3-a7bd2d1fbb71-kube-api-access-d728t\") pod \"premium-w1-f276c3-kserve-75c9f975b8-z2g26\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:40.960389 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.960324 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-dshm\") pod \"premium-w1-f276c3-kserve-75c9f975b8-z2g26\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:40.960547 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.960471 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-home\") pod \"premium-w1-f276c3-kserve-75c9f975b8-z2g26\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:40.960598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.960548 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-tmp-dir\") pod \"premium-w1-f276c3-kserve-75c9f975b8-z2g26\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:40.960663 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.960618 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-model-cache\") pod \"premium-w1-f276c3-kserve-75c9f975b8-z2g26\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:40.960842 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.960817 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-home\") pod \"premium-w1-f276c3-kserve-75c9f975b8-z2g26\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:40.962547 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.962528 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-dshm\") pod \"premium-w1-f276c3-kserve-75c9f975b8-z2g26\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:40.962772 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:40.962756 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e539f99-221e-4050-87f3-a7bd2d1fbb71-tls-certs\") pod \"premium-w1-f276c3-kserve-75c9f975b8-z2g26\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:41.022377 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:41.022314 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d728t\" (UniqueName: \"kubernetes.io/projected/4e539f99-221e-4050-87f3-a7bd2d1fbb71-kube-api-access-d728t\") pod \"premium-w1-f276c3-kserve-75c9f975b8-z2g26\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:41.090514 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:41.090483 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:41.471723 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:41.471697 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26"] Nov 02 01:40:41.472952 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:40:41.472929 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e539f99_221e_4050_87f3_a7bd2d1fbb71.slice/crio-08bb363191f2186441c941f17a0cfdf772b44820894120648cfda9e71e1065e7 WatchSource:0}: Error finding container 08bb363191f2186441c941f17a0cfdf772b44820894120648cfda9e71e1065e7: Status 404 returned error can't find the container with id 08bb363191f2186441c941f17a0cfdf772b44820894120648cfda9e71e1065e7 Nov 02 01:40:42.253887 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:42.253850 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" event={"ID":"4e539f99-221e-4050-87f3-a7bd2d1fbb71","Type":"ContainerStarted","Data":"d5bc98f234a9b3c1d46aa172b8418d023df3fd1eadb6ff747a19890a00b06466"} Nov 02 01:40:42.253887 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:42.253892 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" event={"ID":"4e539f99-221e-4050-87f3-a7bd2d1fbb71","Type":"ContainerStarted","Data":"08bb363191f2186441c941f17a0cfdf772b44820894120648cfda9e71e1065e7"} Nov 02 01:40:42.375845 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:42.375783 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podStartSLOduration=2.375765382 podStartE2EDuration="2.375765382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:40:42.371909218 +0000 UTC m=+959.043161901" watchObservedRunningTime="2026-11-02 01:40:42.375765382 +0000 UTC m=+959.047018065" Nov 02 01:40:43.090654 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:43.090616 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:43.962360 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:43.962324 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:40:48.774903 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:48.774870 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:40:51.090729 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:51.090693 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:51.106290 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:51.106258 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:51.321203 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:51.321173 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:40:51.738344 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:51.738305 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:40:53.585324 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:53.585293 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:40:54.172700 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:40:54.172641 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:40:58.374778 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:40:58.374744 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:41:03.100788 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.100738 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:41:03.108155 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.108125 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj"] Nov 02 01:41:03.111902 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.111880 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:03.192204 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.192170 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-model-cache\") pod \"premium-w0-b71ba5-kserve-5594fffdd8-nx7tj\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:03.192392 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.192231 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvf7s\" (UniqueName: \"kubernetes.io/projected/cada31e5-25a0-464e-9e89-f857de57beba-kube-api-access-fvf7s\") pod \"premium-w0-b71ba5-kserve-5594fffdd8-nx7tj\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:03.192392 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.192256 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-dshm\") pod \"premium-w0-b71ba5-kserve-5594fffdd8-nx7tj\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:03.192392 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.192360 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-home\") pod \"premium-w0-b71ba5-kserve-5594fffdd8-nx7tj\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:03.192571 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.192497 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cada31e5-25a0-464e-9e89-f857de57beba-tls-certs\") pod \"premium-w0-b71ba5-kserve-5594fffdd8-nx7tj\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:03.192571 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.192554 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-tmp-dir\") pod \"premium-w0-b71ba5-kserve-5594fffdd8-nx7tj\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:03.215937 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.215901 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b71ba5\"/\"premium-w0-b71ba5-kserve-self-signed-certs\"" Nov 02 01:41:03.218704 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.218654 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj"] Nov 02 01:41:03.240030 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.240005 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:41:03.293704 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.293646 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cada31e5-25a0-464e-9e89-f857de57beba-tls-certs\") pod \"premium-w0-b71ba5-kserve-5594fffdd8-nx7tj\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:03.293873 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.293717 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-tmp-dir\") pod \"premium-w0-b71ba5-kserve-5594fffdd8-nx7tj\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:03.293873 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.293751 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-model-cache\") pod \"premium-w0-b71ba5-kserve-5594fffdd8-nx7tj\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:03.293873 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.293793 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvf7s\" (UniqueName: \"kubernetes.io/projected/cada31e5-25a0-464e-9e89-f857de57beba-kube-api-access-fvf7s\") pod \"premium-w0-b71ba5-kserve-5594fffdd8-nx7tj\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:03.293873 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.293812 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-dshm\") pod \"premium-w0-b71ba5-kserve-5594fffdd8-nx7tj\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:03.293873 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.293829 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-home\") pod \"premium-w0-b71ba5-kserve-5594fffdd8-nx7tj\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:03.294178 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.294152 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-model-cache\") pod \"premium-w0-b71ba5-kserve-5594fffdd8-nx7tj\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:03.294235 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.294151 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-tmp-dir\") pod \"premium-w0-b71ba5-kserve-5594fffdd8-nx7tj\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:03.294235 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.294219 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-home\") pod \"premium-w0-b71ba5-kserve-5594fffdd8-nx7tj\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:03.296010 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.295984 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-dshm\") pod \"premium-w0-b71ba5-kserve-5594fffdd8-nx7tj\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:03.296183 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.296167 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cada31e5-25a0-464e-9e89-f857de57beba-tls-certs\") pod \"premium-w0-b71ba5-kserve-5594fffdd8-nx7tj\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:03.356227 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.356151 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvf7s\" (UniqueName: \"kubernetes.io/projected/cada31e5-25a0-464e-9e89-f857de57beba-kube-api-access-fvf7s\") pod \"premium-w0-b71ba5-kserve-5594fffdd8-nx7tj\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:03.424201 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.424168 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:03.828483 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:03.828445 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj"] Nov 02 01:41:03.830218 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:41:03.830188 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcada31e5_25a0_464e_9e89_f857de57beba.slice/crio-d148ee7b116c804c71c73aba4c2f6237a6d2bb10363f62e60b3d442b1b0d64aa WatchSource:0}: Error finding container d148ee7b116c804c71c73aba4c2f6237a6d2bb10363f62e60b3d442b1b0d64aa: Status 404 returned error can't find the container with id d148ee7b116c804c71c73aba4c2f6237a6d2bb10363f62e60b3d442b1b0d64aa Nov 02 01:41:04.390808 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:04.390768 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" event={"ID":"cada31e5-25a0-464e-9e89-f857de57beba","Type":"ContainerStarted","Data":"d09c6b30ab2c39f70d1a48ed9474712b5f4699522d52ff746e00fa5426c34787"} Nov 02 01:41:04.391210 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:04.390821 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" event={"ID":"cada31e5-25a0-464e-9e89-f857de57beba","Type":"ContainerStarted","Data":"d148ee7b116c804c71c73aba4c2f6237a6d2bb10363f62e60b3d442b1b0d64aa"} Nov 02 01:41:04.425216 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:04.425193 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:04.510796 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:04.510744 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podStartSLOduration=1.510730651 podStartE2EDuration="1.510730651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:41:04.509142074 +0000 UTC m=+981.180394754" watchObservedRunningTime="2026-11-02 01:41:04.510730651 +0000 UTC m=+981.181983332" Nov 02 01:41:07.837262 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:07.837232 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:41:12.672906 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:12.672840 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:41:13.424843 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:13.424807 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:13.438123 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:13.438099 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:13.453513 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:13.453488 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:41:17.388689 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:17.388637 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:41:18.425434 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.425401 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22"] Nov 02 01:41:18.429615 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.429599 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:18.462496 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.462467 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1c8fde-4ec7-433a-9f68-0470dee95c46-tls-certs\") pod \"premium-w4-187478-kserve-6496c99c69-fdv22\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:18.462657 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.462633 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-tmp-dir\") pod \"premium-w4-187478-kserve-6496c99c69-fdv22\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:18.462746 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.462718 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-model-cache\") pod \"premium-w4-187478-kserve-6496c99c69-fdv22\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:18.462827 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.462812 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-dshm\") pod \"premium-w4-187478-kserve-6496c99c69-fdv22\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:18.462890 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.462853 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-home\") pod \"premium-w4-187478-kserve-6496c99c69-fdv22\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:18.462890 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.462882 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6kll\" (UniqueName: \"kubernetes.io/projected/9a1c8fde-4ec7-433a-9f68-0470dee95c46-kube-api-access-s6kll\") pod \"premium-w4-187478-kserve-6496c99c69-fdv22\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:18.486611 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.486585 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22"] Nov 02 01:41:18.513850 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.513822 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:41:18.534690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.534656 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-187478\"/\"openshift-service-ca.crt\"" Nov 02 01:41:18.534802 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.534729 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-187478\"/\"kube-root-ca.crt\"" Nov 02 01:41:18.537099 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.537085 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-187478\"/\"default-dockercfg-qnkcp\"" Nov 02 01:41:18.537260 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.537247 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-187478\"/\"premium-w4-187478-kserve-self-signed-certs\"" Nov 02 01:41:18.564042 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.564013 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-tmp-dir\") pod \"premium-w4-187478-kserve-6496c99c69-fdv22\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:18.564194 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.564054 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-model-cache\") pod \"premium-w4-187478-kserve-6496c99c69-fdv22\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:18.564194 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.564148 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-dshm\") pod \"premium-w4-187478-kserve-6496c99c69-fdv22\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:18.564194 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.564185 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-home\") pod \"premium-w4-187478-kserve-6496c99c69-fdv22\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:18.564373 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.564215 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6kll\" (UniqueName: \"kubernetes.io/projected/9a1c8fde-4ec7-433a-9f68-0470dee95c46-kube-api-access-s6kll\") pod \"premium-w4-187478-kserve-6496c99c69-fdv22\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:18.564373 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.564295 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1c8fde-4ec7-433a-9f68-0470dee95c46-tls-certs\") pod \"premium-w4-187478-kserve-6496c99c69-fdv22\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:18.564493 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.564470 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-tmp-dir\") pod \"premium-w4-187478-kserve-6496c99c69-fdv22\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:18.565054 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.565029 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-home\") pod \"premium-w4-187478-kserve-6496c99c69-fdv22\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:18.565242 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.565212 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-model-cache\") pod \"premium-w4-187478-kserve-6496c99c69-fdv22\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:18.567197 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.567169 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-dshm\") pod \"premium-w4-187478-kserve-6496c99c69-fdv22\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:18.568144 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.568118 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1c8fde-4ec7-433a-9f68-0470dee95c46-tls-certs\") pod \"premium-w4-187478-kserve-6496c99c69-fdv22\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:18.646507 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.646465 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6kll\" (UniqueName: \"kubernetes.io/projected/9a1c8fde-4ec7-433a-9f68-0470dee95c46-kube-api-access-s6kll\") pod \"premium-w4-187478-kserve-6496c99c69-fdv22\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:18.742454 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:18.742376 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:19.130806 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:19.130781 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22"] Nov 02 01:41:19.131855 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:41:19.131828 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a1c8fde_4ec7_433a_9f68_0470dee95c46.slice/crio-9e296d4a87d85e0c2edf65b9958f192ad14ce4fbf9782d5d1624335f54ac65aa WatchSource:0}: Error finding container 9e296d4a87d85e0c2edf65b9958f192ad14ce4fbf9782d5d1624335f54ac65aa: Status 404 returned error can't find the container with id 9e296d4a87d85e0c2edf65b9958f192ad14ce4fbf9782d5d1624335f54ac65aa Nov 02 01:41:19.476942 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:19.476865 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" event={"ID":"9a1c8fde-4ec7-433a-9f68-0470dee95c46","Type":"ContainerStarted","Data":"c8d360b801611315f2ad338dd55c5dad45bdd99cf73c7fa0f556de4a3ddeb8a6"} Nov 02 01:41:19.476942 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:19.476898 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" event={"ID":"9a1c8fde-4ec7-433a-9f68-0470dee95c46","Type":"ContainerStarted","Data":"9e296d4a87d85e0c2edf65b9958f192ad14ce4fbf9782d5d1624335f54ac65aa"} Nov 02 01:41:19.599337 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:19.599281 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podStartSLOduration=1.599268584 podStartE2EDuration="1.599268584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:41:19.596873308 +0000 UTC m=+996.268125989" watchObservedRunningTime="2026-11-02 01:41:19.599268584 +0000 UTC m=+996.270521266" Nov 02 01:41:19.743042 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:19.742964 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:22.173822 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:22.173787 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:41:24.172608 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:41:24.172572 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:41:27.015112 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:27.015080 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:41:28.743315 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:28.743281 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:28.755810 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:28.755782 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:29.551752 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:29.551723 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:41:31.798180 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:31.798143 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:41:36.578931 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:36.578900 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:41:36.770717 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:36.770664 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:41:38.301390 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.301358 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb"] Nov 02 01:41:38.305414 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.305396 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.406316 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.405917 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.406316 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.405976 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.406316 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.406034 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07be5104-2adc-4dff-87bb-f115473caa5d-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.406316 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.406079 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.406316 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.406134 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07be5104-2adc-4dff-87bb-f115473caa5d-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.406316 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.406215 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.406316 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.406264 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb4zl\" (UniqueName: \"kubernetes.io/projected/07be5104-2adc-4dff-87bb-f115473caa5d-kube-api-access-xb4zl\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.406805 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.406417 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.406805 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.406508 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07be5104-2adc-4dff-87bb-f115473caa5d-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.409323 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.409293 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-36a1e0-openshift-default-dockercfg-lck7x\"" Nov 02 01:41:38.413279 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.413250 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb"] Nov 02 01:41:38.507326 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.507294 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.507508 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.507344 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07be5104-2adc-4dff-87bb-f115473caa5d-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.507508 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.507386 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.507508 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.507434 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07be5104-2adc-4dff-87bb-f115473caa5d-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.507508 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.507505 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.507764 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.507553 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb4zl\" (UniqueName: \"kubernetes.io/projected/07be5104-2adc-4dff-87bb-f115473caa5d-kube-api-access-xb4zl\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.507764 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.507604 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.507764 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.507652 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07be5104-2adc-4dff-87bb-f115473caa5d-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.507764 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.507718 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.507965 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.507792 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.507965 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.507856 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.508067 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.508007 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.508116 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.508092 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.508256 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.508234 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07be5104-2adc-4dff-87bb-f115473caa5d-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.509837 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.509815 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.510103 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.510085 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07be5104-2adc-4dff-87bb-f115473caa5d-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.569186 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.569121 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07be5104-2adc-4dff-87bb-f115473caa5d-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.569376 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.569353 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb4zl\" (UniqueName: \"kubernetes.io/projected/07be5104-2adc-4dff-87bb-f115473caa5d-kube-api-access-xb4zl\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:38.594339 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.594313 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:41:38.623592 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:38.623568 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:39.051762 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:39.051735 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb"] Nov 02 01:41:39.052720 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:41:39.052692 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07be5104_2adc_4dff_87bb_f115473caa5d.slice/crio-b0a7c6e64be5dcc46dea523b88cb19aca6f745517744a61eee39f402faaae8ff WatchSource:0}: Error finding container b0a7c6e64be5dcc46dea523b88cb19aca6f745517744a61eee39f402faaae8ff: Status 404 returned error can't find the container with id b0a7c6e64be5dcc46dea523b88cb19aca6f745517744a61eee39f402faaae8ff Nov 02 01:41:39.055463 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:39.055433 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:41:39.055548 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:39.055497 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:41:39.055548 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:39.055526 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:41:39.606619 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:39.606583 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" event={"ID":"07be5104-2adc-4dff-87bb-f115473caa5d","Type":"ContainerStarted","Data":"89054247c845b795c849c3db9c9213c741e423afce3a4f04bf7aacd55c2cf363"} Nov 02 01:41:39.607060 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:39.606648 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" event={"ID":"07be5104-2adc-4dff-87bb-f115473caa5d","Type":"ContainerStarted","Data":"b0a7c6e64be5dcc46dea523b88cb19aca6f745517744a61eee39f402faaae8ff"} Nov 02 01:41:39.623640 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:39.623616 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:39.728272 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:39.728215 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" podStartSLOduration=1.728202649 podStartE2EDuration="1.728202649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:41:39.727777086 +0000 UTC m=+1016.399029768" watchObservedRunningTime="2026-11-02 01:41:39.728202649 +0000 UTC m=+1016.399455329" Nov 02 01:41:40.625778 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:40.625747 5044 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 02 01:41:40.626241 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:40.625805 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" podUID="07be5104-2adc-4dff-87bb-f115473caa5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 02 01:41:41.386553 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:41.386518 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:41:41.623922 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:41.623888 5044 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 02 01:41:41.624099 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:41.623945 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" podUID="07be5104-2adc-4dff-87bb-f115473caa5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 02 01:41:42.623945 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:42.623907 5044 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 02 01:41:42.624419 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:42.623968 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" podUID="07be5104-2adc-4dff-87bb-f115473caa5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 02 01:41:43.624195 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:43.624144 5044 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 02 01:41:43.624728 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:43.624219 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" podUID="07be5104-2adc-4dff-87bb-f115473caa5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 02 01:41:44.624165 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:44.624130 5044 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Nov 02 01:41:44.624313 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:44.624182 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" podUID="07be5104-2adc-4dff-87bb-f115473caa5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Nov 02 01:41:46.023252 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:46.023208 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:46.024022 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:46.023999 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:46.026719 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:46.026697 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:41:46.262820 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:46.262779 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:41:50.983110 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:50.983076 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:41:54.172597 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:41:54.172511 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:41:55.716444 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:55.716386 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:41:56.235436 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:56.235403 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps"] Nov 02 01:41:56.235696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:41:56.235644 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" podUID="82fc1a10-7f06-4790-94bd-c57c06c06803" containerName="istio-proxy" containerID="cri-o://d47aea04e42b19bd5c06e77bc6c9d48280cb032c75df7f1b668cba60d7030155" gracePeriod=30 Nov 02 01:42:00.491637 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:00.491593 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:01.506309 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.506251 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:42:01.582820 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.582789 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-credential-socket\") pod \"82fc1a10-7f06-4790-94bd-c57c06c06803\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " Nov 02 01:42:01.583013 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.582840 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82fc1a10-7f06-4790-94bd-c57c06c06803-kube-api-access-qn9c4\" (UniqueName: \"kubernetes.io/projected/82fc1a10-7f06-4790-94bd-c57c06c06803-kube-api-access-qn9c4\") pod \"82fc1a10-7f06-4790-94bd-c57c06c06803\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " Nov 02 01:42:01.583013 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.582882 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-data\") pod \"82fc1a10-7f06-4790-94bd-c57c06c06803\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " Nov 02 01:42:01.583013 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.582954 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-envoy\") pod \"82fc1a10-7f06-4790-94bd-c57c06c06803\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " Nov 02 01:42:01.583013 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.582984 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-token\" (UniqueName: \"kubernetes.io/projected/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-token\") pod \"82fc1a10-7f06-4790-94bd-c57c06c06803\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " Nov 02 01:42:01.583241 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.583013 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-workload-certs\") pod \"82fc1a10-7f06-4790-94bd-c57c06c06803\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " Nov 02 01:42:01.583241 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.583085 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-workload-socket\") pod \"82fc1a10-7f06-4790-94bd-c57c06c06803\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " Nov 02 01:42:01.583241 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.583116 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-podinfo\") pod \"82fc1a10-7f06-4790-94bd-c57c06c06803\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " Nov 02 01:42:01.583241 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.583114 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-credential-socket" pod "82fc1a10-7f06-4790-94bd-c57c06c06803" (UID: "82fc1a10-7f06-4790-94bd-c57c06c06803"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:01.583241 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.583187 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82fc1a10-7f06-4790-94bd-c57c06c06803-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82fc1a10-7f06-4790-94bd-c57c06c06803-istiod-ca-cert\") pod \"82fc1a10-7f06-4790-94bd-c57c06c06803\" (UID: \"82fc1a10-7f06-4790-94bd-c57c06c06803\") " Nov 02 01:42:01.583499 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.583244 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-data" pod "82fc1a10-7f06-4790-94bd-c57c06c06803" (UID: "82fc1a10-7f06-4790-94bd-c57c06c06803"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:01.583558 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.583486 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-workload-socket" pod "82fc1a10-7f06-4790-94bd-c57c06c06803" (UID: "82fc1a10-7f06-4790-94bd-c57c06c06803"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:01.583717 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.583693 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-workload-certs" pod "82fc1a10-7f06-4790-94bd-c57c06c06803" (UID: "82fc1a10-7f06-4790-94bd-c57c06c06803"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:01.583794 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.583628 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82fc1a10-7f06-4790-94bd-c57c06c06803-istiod-ca-cert" pod "82fc1a10-7f06-4790-94bd-c57c06c06803" (UID: "82fc1a10-7f06-4790-94bd-c57c06c06803"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:42:01.583794 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.583719 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:42:01.583794 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.583754 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:42:01.583794 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.583770 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:42:01.585802 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.585774 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-envoy" pod "82fc1a10-7f06-4790-94bd-c57c06c06803" (UID: "82fc1a10-7f06-4790-94bd-c57c06c06803"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:42:01.585922 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.585799 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82fc1a10-7f06-4790-94bd-c57c06c06803-kube-api-access-qn9c4" pod "82fc1a10-7f06-4790-94bd-c57c06c06803" (UID: "82fc1a10-7f06-4790-94bd-c57c06c06803"). InnerVolumeSpecName "kube-api-access-qn9c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:42:01.585922 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.585892 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-podinfo" pod "82fc1a10-7f06-4790-94bd-c57c06c06803" (UID: "82fc1a10-7f06-4790-94bd-c57c06c06803"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:42:01.585922 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.585904 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-token" pod "82fc1a10-7f06-4790-94bd-c57c06c06803" (UID: "82fc1a10-7f06-4790-94bd-c57c06c06803"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:42:01.686315 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.684640 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:42:01.686315 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.684709 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:42:01.686315 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.684729 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82fc1a10-7f06-4790-94bd-c57c06c06803-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:42:01.686315 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.684746 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82fc1a10-7f06-4790-94bd-c57c06c06803-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:42:01.686315 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.684762 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82fc1a10-7f06-4790-94bd-c57c06c06803-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:42:01.686315 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.684779 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn9c4\" (UniqueName: \"kubernetes.io/projected/82fc1a10-7f06-4790-94bd-c57c06c06803-kube-api-access-qn9c4\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:42:01.751227 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.751194 5044 generic.go:356] "Generic (PLEG): container finished" podID="82fc1a10-7f06-4790-94bd-c57c06c06803" containerID="d47aea04e42b19bd5c06e77bc6c9d48280cb032c75df7f1b668cba60d7030155" exitCode=0 Nov 02 01:42:01.751377 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.751236 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" event={"ID":"82fc1a10-7f06-4790-94bd-c57c06c06803","Type":"ContainerDied","Data":"d47aea04e42b19bd5c06e77bc6c9d48280cb032c75df7f1b668cba60d7030155"} Nov 02 01:42:01.751377 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.751279 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" event={"ID":"82fc1a10-7f06-4790-94bd-c57c06c06803","Type":"ContainerDied","Data":"69420e77b9e0f4f42858ebc036fab8602c122d9d0b21bfb450d8952d5830dce7"} Nov 02 01:42:01.751377 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.751300 5044 scope.go:122] "RemoveContainer" containerID="d47aea04e42b19bd5c06e77bc6c9d48280cb032c75df7f1b668cba60d7030155" Nov 02 01:42:01.751377 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.751238 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps" Nov 02 01:42:01.762086 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.762068 5044 scope.go:122] "RemoveContainer" containerID="d47aea04e42b19bd5c06e77bc6c9d48280cb032c75df7f1b668cba60d7030155" Nov 02 01:42:01.762393 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:42:01.762375 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47aea04e42b19bd5c06e77bc6c9d48280cb032c75df7f1b668cba60d7030155\": container with ID starting with d47aea04e42b19bd5c06e77bc6c9d48280cb032c75df7f1b668cba60d7030155 not found: ID does not exist" containerID="d47aea04e42b19bd5c06e77bc6c9d48280cb032c75df7f1b668cba60d7030155" Nov 02 01:42:01.762447 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.762404 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47aea04e42b19bd5c06e77bc6c9d48280cb032c75df7f1b668cba60d7030155"} err="failed to get container status \"d47aea04e42b19bd5c06e77bc6c9d48280cb032c75df7f1b668cba60d7030155\": rpc error: code = NotFound desc = could not find container \"d47aea04e42b19bd5c06e77bc6c9d48280cb032c75df7f1b668cba60d7030155\": container with ID starting with d47aea04e42b19bd5c06e77bc6c9d48280cb032c75df7f1b668cba60d7030155 not found: ID does not exist" Nov 02 01:42:01.919339 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.919305 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps"] Nov 02 01:42:01.923548 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:01.923520 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-cj4ps"] Nov 02 01:42:03.454922 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:03.454881 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82fc1a10-7f06-4790-94bd-c57c06c06803" path="/var/lib/kubelet/pods/82fc1a10-7f06-4790-94bd-c57c06c06803/volumes" Nov 02 01:42:05.371062 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:05.371030 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:10.217291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:10.217259 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:15.080219 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:15.080184 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:16.017240 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:16.017200 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:16.165424 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:16.165388 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:19.875425 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:19.875392 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:23.265102 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.265067 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp"] Nov 02 01:42:23.265627 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.265613 5044 cpu_manager.go:414] "containerMap: removing container" podUID="82fc1a10-7f06-4790-94bd-c57c06c06803" containerName="istio-proxy" Nov 02 01:42:23.265688 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.265628 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82fc1a10-7f06-4790-94bd-c57c06c06803" containerName="istio-proxy" Nov 02 01:42:23.265739 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.265729 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="82fc1a10-7f06-4790-94bd-c57c06c06803" containerName="istio-proxy" Nov 02 01:42:23.270665 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.270640 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.325937 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.325912 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-88e76d-openshift-default-dockercfg-gsdsg\"" Nov 02 01:42:23.345210 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.345179 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.345348 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.345298 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.345348 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.345331 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.345443 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.345394 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce5a9bbf-1908-4220-b871-452bac6bebb8-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.345579 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.345554 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.345626 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.345600 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.345777 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.345641 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.345860 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.345802 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz7l2\" (UniqueName: \"kubernetes.io/projected/ce5a9bbf-1908-4220-b871-452bac6bebb8-kube-api-access-pz7l2\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.345913 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.345861 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.379291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.379270 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp"] Nov 02 01:42:23.447362 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.447323 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.447525 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.447371 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.447525 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.447411 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce5a9bbf-1908-4220-b871-452bac6bebb8-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.447525 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.447465 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.447525 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.447495 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.447803 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.447529 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.447803 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.447586 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz7l2\" (UniqueName: \"kubernetes.io/projected/ce5a9bbf-1908-4220-b871-452bac6bebb8-kube-api-access-pz7l2\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.447803 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.447630 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.447803 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.447750 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.447803 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.447780 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.448361 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.448339 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce5a9bbf-1908-4220-b871-452bac6bebb8-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.448518 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.448360 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.448625 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.448597 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.448701 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.448556 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.450587 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.450569 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.450820 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.450803 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.510140 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.510105 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.510275 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.510218 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz7l2\" (UniqueName: \"kubernetes.io/projected/ce5a9bbf-1908-4220-b871-452bac6bebb8-kube-api-access-pz7l2\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:23.584719 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:23.584633 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:24.021234 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:24.021212 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp"] Nov 02 01:42:24.022163 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:42:24.022140 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce5a9bbf_1908_4220_b871_452bac6bebb8.slice/crio-29dcdd94198eeab0dfdda4d2405bfb355a87b578d55e0fd377132db3bb8c2c09 WatchSource:0}: Error finding container 29dcdd94198eeab0dfdda4d2405bfb355a87b578d55e0fd377132db3bb8c2c09: Status 404 returned error can't find the container with id 29dcdd94198eeab0dfdda4d2405bfb355a87b578d55e0fd377132db3bb8c2c09 Nov 02 01:42:24.024595 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:24.024565 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:42:24.024705 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:24.024638 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:42:24.024705 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:24.024666 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:42:24.175585 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:42:24.174534 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:42:24.688236 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:24.688177 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:24.919174 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:24.919132 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" event={"ID":"ce5a9bbf-1908-4220-b871-452bac6bebb8","Type":"ContainerStarted","Data":"de1c9b12ae2050bdc1a6d7db5c9da8477ae71fd00d506495d5c1e7885f40cc1e"} Nov 02 01:42:24.919369 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:24.919214 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" event={"ID":"ce5a9bbf-1908-4220-b871-452bac6bebb8","Type":"ContainerStarted","Data":"29dcdd94198eeab0dfdda4d2405bfb355a87b578d55e0fd377132db3bb8c2c09"} Nov 02 01:42:25.153971 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:25.153916 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" podStartSLOduration=2.153900653 podStartE2EDuration="2.153900653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:42:25.10112004 +0000 UTC m=+1061.772372721" watchObservedRunningTime="2026-11-02 01:42:25.153900653 +0000 UTC m=+1061.825153333" Nov 02 01:42:25.587844 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:25.587795 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:25.589933 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:25.589905 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 02 01:42:25.590138 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:25.589954 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" podUID="ce5a9bbf-1908-4220-b871-452bac6bebb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 02 01:42:26.586037 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:26.586001 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 02 01:42:26.586533 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:26.586065 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" podUID="ce5a9bbf-1908-4220-b871-452bac6bebb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 02 01:42:27.586159 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:27.586117 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 02 01:42:27.586706 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:27.586193 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" podUID="ce5a9bbf-1908-4220-b871-452bac6bebb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 02 01:42:28.588695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:28.588117 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 02 01:42:28.588695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:28.588177 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" podUID="ce5a9bbf-1908-4220-b871-452bac6bebb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 02 01:42:29.194153 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:29.194120 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:29.486345 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:29.486254 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:29.585696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:29.585078 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 02 01:42:29.585696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:29.585151 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" podUID="ce5a9bbf-1908-4220-b871-452bac6bebb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 02 01:42:30.695736 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:30.695701 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:30.696531 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:30.696422 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:30.697646 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:30.697626 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:42:34.210529 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:34.210498 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:38.930512 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:38.930478 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:39.073023 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:39.072983 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:43.885656 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:43.885569 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:44.063011 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:44.062976 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:44.203987 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:44.203909 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:44.918370 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:44.918340 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:45.059941 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:45.059910 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:48.617077 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:48.617042 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:53.466165 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:53.466130 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:54.177809 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:42:54.177768 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:42:58.232361 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:58.232317 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:42:59.724700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:59.724656 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7"] Nov 02 01:42:59.728556 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:59.728541 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7" Nov 02 01:42:59.798609 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:59.798579 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-634cc270-z88h7\" (UID: \"c6bcae83-1781-4dd1-bca6-fcd63d380dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7" Nov 02 01:42:59.798839 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:59.798728 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-634cc270-z88h7\" (UID: \"c6bcae83-1781-4dd1-bca6-fcd63d380dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7" Nov 02 01:42:59.798839 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:59.798788 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmfw9\" (UniqueName: \"kubernetes.io/projected/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-kube-api-access-tmfw9\") pod \"maas-api-revoke-sub-634cc270-z88h7\" (UID: \"c6bcae83-1781-4dd1-bca6-fcd63d380dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7" Nov 02 01:42:59.833204 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:59.833177 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:42:59.833353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:59.833242 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\"" Nov 02 01:42:59.836042 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:59.836013 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7"] Nov 02 01:42:59.900057 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:59.900026 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmfw9\" (UniqueName: \"kubernetes.io/projected/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-kube-api-access-tmfw9\") pod \"maas-api-revoke-sub-634cc270-z88h7\" (UID: \"c6bcae83-1781-4dd1-bca6-fcd63d380dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7" Nov 02 01:42:59.900213 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:59.900114 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-634cc270-z88h7\" (UID: \"c6bcae83-1781-4dd1-bca6-fcd63d380dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7" Nov 02 01:42:59.900213 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:59.900186 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-634cc270-z88h7\" (UID: \"c6bcae83-1781-4dd1-bca6-fcd63d380dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7" Nov 02 01:42:59.900833 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:59.900811 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-634cc270-z88h7\" (UID: \"c6bcae83-1781-4dd1-bca6-fcd63d380dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7" Nov 02 01:42:59.902298 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:59.902267 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-634cc270-z88h7\" (UID: \"c6bcae83-1781-4dd1-bca6-fcd63d380dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7" Nov 02 01:42:59.961719 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:42:59.961686 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmfw9\" (UniqueName: \"kubernetes.io/projected/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-kube-api-access-tmfw9\") pod \"maas-api-revoke-sub-634cc270-z88h7\" (UID: \"c6bcae83-1781-4dd1-bca6-fcd63d380dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7" Nov 02 01:43:00.041888 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:00.041854 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7" Nov 02 01:43:00.424971 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:00.424945 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7"] Nov 02 01:43:00.426369 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:43:00.426345 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6bcae83_1781_4dd1_bca6_fcd63d380dfa.slice/crio-2efd7bba7af8d7b7769d28ea77a2122a4bbb4388d48801d243c98f4e3ede3564 WatchSource:0}: Error finding container 2efd7bba7af8d7b7769d28ea77a2122a4bbb4388d48801d243c98f4e3ede3564: Status 404 returned error can't find the container with id 2efd7bba7af8d7b7769d28ea77a2122a4bbb4388d48801d243c98f4e3ede3564 Nov 02 01:43:00.428414 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:00.428396 5044 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:43:01.137165 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:01.137127 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7" event={"ID":"c6bcae83-1781-4dd1-bca6-fcd63d380dfa","Type":"ContainerStarted","Data":"2efd7bba7af8d7b7769d28ea77a2122a4bbb4388d48801d243c98f4e3ede3564"} Nov 02 01:43:02.801016 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:02.800975 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:43:03.113240 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:03.113148 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:43:07.998321 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:07.998287 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:43:09.195051 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:09.195016 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7" event={"ID":"c6bcae83-1781-4dd1-bca6-fcd63d380dfa","Type":"ContainerStarted","Data":"a928a201d99ab8682eabfb09e1d52bf1ba1256a6ada23d98ff559b31a6d926f8"} Nov 02 01:43:09.316635 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:09.316570 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7" podStartSLOduration=1.838323666 podStartE2EDuration="10.316550976s" totalImagesPullingTime="8.47822731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:09.314055505 +0000 UTC m=+1105.985308188" watchObservedRunningTime="2026-11-02 01:43:09.316550976 +0000 UTC m=+1105.987803660" Nov 02 01:43:10.202285 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:10.202190 5044 generic.go:356] "Generic (PLEG): container finished" podID="c6bcae83-1781-4dd1-bca6-fcd63d380dfa" containerID="a928a201d99ab8682eabfb09e1d52bf1ba1256a6ada23d98ff559b31a6d926f8" exitCode=0 Nov 02 01:43:10.202646 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:10.202286 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7" event={"ID":"c6bcae83-1781-4dd1-bca6-fcd63d380dfa","Type":"ContainerDied","Data":"a928a201d99ab8682eabfb09e1d52bf1ba1256a6ada23d98ff559b31a6d926f8"} Nov 02 01:43:11.361731 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:11.361706 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7" Nov 02 01:43:11.428790 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:11.428762 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-maas-api-xks-ca\") pod \"c6bcae83-1781-4dd1-bca6-fcd63d380dfa\" (UID: \"c6bcae83-1781-4dd1-bca6-fcd63d380dfa\") " Nov 02 01:43:11.428923 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:11.428821 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-kube-api-access-tmfw9\" (UniqueName: \"kubernetes.io/projected/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-kube-api-access-tmfw9\") pod \"c6bcae83-1781-4dd1-bca6-fcd63d380dfa\" (UID: \"c6bcae83-1781-4dd1-bca6-fcd63d380dfa\") " Nov 02 01:43:11.428923 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:11.428845 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-maas-api-openshift-ca\") pod \"c6bcae83-1781-4dd1-bca6-fcd63d380dfa\" (UID: \"c6bcae83-1781-4dd1-bca6-fcd63d380dfa\") " Nov 02 01:43:11.429338 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:11.429309 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-maas-api-openshift-ca" pod "c6bcae83-1781-4dd1-bca6-fcd63d380dfa" (UID: "c6bcae83-1781-4dd1-bca6-fcd63d380dfa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:11.431050 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:11.431018 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-kube-api-access-tmfw9" pod "c6bcae83-1781-4dd1-bca6-fcd63d380dfa" (UID: "c6bcae83-1781-4dd1-bca6-fcd63d380dfa"). InnerVolumeSpecName "kube-api-access-tmfw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:11.431153 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:11.431078 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-maas-api-xks-ca" pod "c6bcae83-1781-4dd1-bca6-fcd63d380dfa" (UID: "c6bcae83-1781-4dd1-bca6-fcd63d380dfa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:43:11.529939 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:11.529873 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-maas-api-xks-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:43:11.529939 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:11.529903 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmfw9\" (UniqueName: \"kubernetes.io/projected/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-kube-api-access-tmfw9\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:43:11.529939 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:11.529918 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6bcae83-1781-4dd1-bca6-fcd63d380dfa-maas-api-openshift-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:43:12.216719 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:12.216663 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7" event={"ID":"c6bcae83-1781-4dd1-bca6-fcd63d380dfa","Type":"ContainerDied","Data":"2efd7bba7af8d7b7769d28ea77a2122a4bbb4388d48801d243c98f4e3ede3564"} Nov 02 01:43:12.216883 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:12.216736 5044 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2efd7bba7af8d7b7769d28ea77a2122a4bbb4388d48801d243c98f4e3ede3564" Nov 02 01:43:12.216883 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:12.216661 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7" Nov 02 01:43:12.916440 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:12.916407 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:43:17.048307 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:17.048278 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:43:17.799096 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:17.799062 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:43:22.669395 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:22.669321 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:43:24.175263 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:43:24.175223 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:43:26.946013 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:26.945977 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54"] Nov 02 01:43:26.946499 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:26.946215 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" podUID="832cc120-3c4e-4605-b990-683ca969b6ab" containerName="istio-proxy" containerID="cri-o://36d28374340aeb8a48afc28c28a583d1db1ec6fbcc50db6cb1ad908ea09631b2" gracePeriod=30 Nov 02 01:43:27.479095 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:27.479059 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:43:32.219605 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.219543 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:43:32.305992 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.305960 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:43:32.346195 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.346167 5044 generic.go:356] "Generic (PLEG): container finished" podID="832cc120-3c4e-4605-b990-683ca969b6ab" containerID="36d28374340aeb8a48afc28c28a583d1db1ec6fbcc50db6cb1ad908ea09631b2" exitCode=0 Nov 02 01:43:32.346380 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.346213 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" Nov 02 01:43:32.346380 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.346231 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" event={"ID":"832cc120-3c4e-4605-b990-683ca969b6ab","Type":"ContainerDied","Data":"36d28374340aeb8a48afc28c28a583d1db1ec6fbcc50db6cb1ad908ea09631b2"} Nov 02 01:43:32.346380 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.346283 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54" event={"ID":"832cc120-3c4e-4605-b990-683ca969b6ab","Type":"ContainerDied","Data":"caca70f04d3fd22470e2b6ca52602e75bf6928f3433f710da6336dc3cb38ca77"} Nov 02 01:43:32.346380 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.346310 5044 scope.go:122] "RemoveContainer" containerID="36d28374340aeb8a48afc28c28a583d1db1ec6fbcc50db6cb1ad908ea09631b2" Nov 02 01:43:32.356040 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.356021 5044 scope.go:122] "RemoveContainer" containerID="36d28374340aeb8a48afc28c28a583d1db1ec6fbcc50db6cb1ad908ea09631b2" Nov 02 01:43:32.356308 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:43:32.356287 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d28374340aeb8a48afc28c28a583d1db1ec6fbcc50db6cb1ad908ea09631b2\": container with ID starting with 36d28374340aeb8a48afc28c28a583d1db1ec6fbcc50db6cb1ad908ea09631b2 not found: ID does not exist" containerID="36d28374340aeb8a48afc28c28a583d1db1ec6fbcc50db6cb1ad908ea09631b2" Nov 02 01:43:32.356360 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.356316 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d28374340aeb8a48afc28c28a583d1db1ec6fbcc50db6cb1ad908ea09631b2"} err="failed to get container status \"36d28374340aeb8a48afc28c28a583d1db1ec6fbcc50db6cb1ad908ea09631b2\": rpc error: code = NotFound desc = could not find container \"36d28374340aeb8a48afc28c28a583d1db1ec6fbcc50db6cb1ad908ea09631b2\": container with ID starting with 36d28374340aeb8a48afc28c28a583d1db1ec6fbcc50db6cb1ad908ea09631b2 not found: ID does not exist" Nov 02 01:43:32.383846 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.383819 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-istio-envoy\") pod \"832cc120-3c4e-4605-b990-683ca969b6ab\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " Nov 02 01:43:32.384000 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.383858 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832cc120-3c4e-4605-b990-683ca969b6ab-kube-api-access-kb5cs\" (UniqueName: \"kubernetes.io/projected/832cc120-3c4e-4605-b990-683ca969b6ab-kube-api-access-kb5cs\") pod \"832cc120-3c4e-4605-b990-683ca969b6ab\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " Nov 02 01:43:32.384000 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.383895 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-workload-socket\") pod \"832cc120-3c4e-4605-b990-683ca969b6ab\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " Nov 02 01:43:32.384000 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.383985 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/832cc120-3c4e-4605-b990-683ca969b6ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832cc120-3c4e-4605-b990-683ca969b6ab-istio-podinfo\") pod \"832cc120-3c4e-4605-b990-683ca969b6ab\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " Nov 02 01:43:32.384187 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.384035 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-workload-certs\") pod \"832cc120-3c4e-4605-b990-683ca969b6ab\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " Nov 02 01:43:32.384187 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.384082 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/832cc120-3c4e-4605-b990-683ca969b6ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832cc120-3c4e-4605-b990-683ca969b6ab-istiod-ca-cert\") pod \"832cc120-3c4e-4605-b990-683ca969b6ab\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " Nov 02 01:43:32.384187 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.384128 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-credential-socket\") pod \"832cc120-3c4e-4605-b990-683ca969b6ab\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " Nov 02 01:43:32.384187 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.384156 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-istio-data\") pod \"832cc120-3c4e-4605-b990-683ca969b6ab\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " Nov 02 01:43:32.384393 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.384205 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832cc120-3c4e-4605-b990-683ca969b6ab-istio-token\" (UniqueName: \"kubernetes.io/projected/832cc120-3c4e-4605-b990-683ca969b6ab-istio-token\") pod \"832cc120-3c4e-4605-b990-683ca969b6ab\" (UID: \"832cc120-3c4e-4605-b990-683ca969b6ab\") " Nov 02 01:43:32.385495 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.385467 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-credential-socket" pod "832cc120-3c4e-4605-b990-683ca969b6ab" (UID: "832cc120-3c4e-4605-b990-683ca969b6ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:32.385607 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.385491 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-workload-certs" pod "832cc120-3c4e-4605-b990-683ca969b6ab" (UID: "832cc120-3c4e-4605-b990-683ca969b6ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:32.385607 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.385523 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-istio-data" pod "832cc120-3c4e-4605-b990-683ca969b6ab" (UID: "832cc120-3c4e-4605-b990-683ca969b6ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:32.385848 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.385820 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-workload-socket" pod "832cc120-3c4e-4605-b990-683ca969b6ab" (UID: "832cc120-3c4e-4605-b990-683ca969b6ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:32.385966 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.385940 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/832cc120-3c4e-4605-b990-683ca969b6ab-istiod-ca-cert" pod "832cc120-3c4e-4605-b990-683ca969b6ab" (UID: "832cc120-3c4e-4605-b990-683ca969b6ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:43:32.387656 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.387616 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/832cc120-3c4e-4605-b990-683ca969b6ab-istio-podinfo" pod "832cc120-3c4e-4605-b990-683ca969b6ab" (UID: "832cc120-3c4e-4605-b990-683ca969b6ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:43:32.387842 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.387733 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832cc120-3c4e-4605-b990-683ca969b6ab-istio-token" pod "832cc120-3c4e-4605-b990-683ca969b6ab" (UID: "832cc120-3c4e-4605-b990-683ca969b6ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:32.388159 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.388133 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832cc120-3c4e-4605-b990-683ca969b6ab-kube-api-access-kb5cs" pod "832cc120-3c4e-4605-b990-683ca969b6ab" (UID: "832cc120-3c4e-4605-b990-683ca969b6ab"). InnerVolumeSpecName "kube-api-access-kb5cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:43:32.388325 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.388292 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-istio-envoy" pod "832cc120-3c4e-4605-b990-683ca969b6ab" (UID: "832cc120-3c4e-4605-b990-683ca969b6ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:43:32.485473 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.485395 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832cc120-3c4e-4605-b990-683ca969b6ab-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:43:32.485473 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.485422 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:43:32.485473 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.485432 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb5cs\" (UniqueName: \"kubernetes.io/projected/832cc120-3c4e-4605-b990-683ca969b6ab-kube-api-access-kb5cs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:43:32.485473 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.485442 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:43:32.485473 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.485451 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832cc120-3c4e-4605-b990-683ca969b6ab-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:43:32.485473 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.485459 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:43:32.485473 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.485467 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832cc120-3c4e-4605-b990-683ca969b6ab-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:43:32.485473 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.485475 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:43:32.485855 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.485485 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832cc120-3c4e-4605-b990-683ca969b6ab-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:43:32.769529 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.769498 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54"] Nov 02 01:43:32.820263 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:32.820236 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-w4b54"] Nov 02 01:43:33.456397 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:33.456366 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="832cc120-3c4e-4605-b990-683ca969b6ab" path="/var/lib/kubelet/pods/832cc120-3c4e-4605-b990-683ca969b6ab/volumes" Nov 02 01:43:37.096722 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.096682 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:43:37.577178 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.577145 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g"] Nov 02 01:43:37.578003 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.577982 5044 cpu_manager.go:414] "containerMap: removing container" podUID="c6bcae83-1781-4dd1-bca6-fcd63d380dfa" containerName="revoke-keys" Nov 02 01:43:37.578064 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.578006 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6bcae83-1781-4dd1-bca6-fcd63d380dfa" containerName="revoke-keys" Nov 02 01:43:37.578064 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.578034 5044 cpu_manager.go:414] "containerMap: removing container" podUID="832cc120-3c4e-4605-b990-683ca969b6ab" containerName="istio-proxy" Nov 02 01:43:37.578064 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.578041 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="832cc120-3c4e-4605-b990-683ca969b6ab" containerName="istio-proxy" Nov 02 01:43:37.578189 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.578178 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6bcae83-1781-4dd1-bca6-fcd63d380dfa" containerName="revoke-keys" Nov 02 01:43:37.578272 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.578195 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="832cc120-3c4e-4605-b990-683ca969b6ab" containerName="istio-proxy" Nov 02 01:43:37.583437 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.583415 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.636479 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.636455 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b71ba5-openshift-default-dockercfg-bzhxb\"" Nov 02 01:43:37.692170 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.692142 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g"] Nov 02 01:43:37.746224 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.746189 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2320a74e-3e24-4633-8d10-807bcafda325-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.746224 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.746227 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.746431 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.746249 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2320a74e-3e24-4633-8d10-807bcafda325-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.746431 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.746294 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.746431 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.746339 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2320a74e-3e24-4633-8d10-807bcafda325-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.746587 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.746566 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.746624 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.746602 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.746662 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.746622 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrl22\" (UniqueName: \"kubernetes.io/projected/2320a74e-3e24-4633-8d10-807bcafda325-kube-api-access-jrl22\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.746809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.746787 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.847811 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.847734 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.847811 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.847771 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.847811 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.847789 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrl22\" (UniqueName: \"kubernetes.io/projected/2320a74e-3e24-4633-8d10-807bcafda325-kube-api-access-jrl22\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.848051 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.847841 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.848051 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.847904 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2320a74e-3e24-4633-8d10-807bcafda325-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.848051 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.847935 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.848051 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.847966 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2320a74e-3e24-4633-8d10-807bcafda325-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.848051 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.848004 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.848051 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.848045 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2320a74e-3e24-4633-8d10-807bcafda325-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.848349 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.848158 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.848513 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.848487 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.848573 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.848506 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.848573 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.848541 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.848801 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.848783 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2320a74e-3e24-4633-8d10-807bcafda325-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.850110 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.850094 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.850385 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.850366 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2320a74e-3e24-4633-8d10-807bcafda325-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.909750 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.909719 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2320a74e-3e24-4633-8d10-807bcafda325-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:37.909885 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:37.909786 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrl22\" (UniqueName: \"kubernetes.io/projected/2320a74e-3e24-4633-8d10-807bcafda325-kube-api-access-jrl22\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:38.199867 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.199783 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:38.595943 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.595899 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt"] Nov 02 01:43:38.601920 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.601896 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.635892 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.635869 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g"] Nov 02 01:43:38.637018 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:43:38.636994 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2320a74e_3e24_4633_8d10_807bcafda325.slice/crio-1c7d89690bcb1ddc2028f4213fba253f03e30c33f81d039def8a37f50a441d13 WatchSource:0}: Error finding container 1c7d89690bcb1ddc2028f4213fba253f03e30c33f81d039def8a37f50a441d13: Status 404 returned error can't find the container with id 1c7d89690bcb1ddc2028f4213fba253f03e30c33f81d039def8a37f50a441d13 Nov 02 01:43:38.639570 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.639535 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:43:38.639708 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.639612 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:43:38.639708 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.639655 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:43:38.685917 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.685890 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt"] Nov 02 01:43:38.760172 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.760126 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.760347 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.760273 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c547c6-30fd-43be-ac27-299da9065a25-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.760347 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.760318 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.760455 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.760364 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbm26\" (UniqueName: \"kubernetes.io/projected/54c547c6-30fd-43be-ac27-299da9065a25-kube-api-access-rbm26\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.760513 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.760494 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c547c6-30fd-43be-ac27-299da9065a25-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.760566 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.760526 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.760618 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.760590 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.760665 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.760641 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.760747 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.760666 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c547c6-30fd-43be-ac27-299da9065a25-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.862430 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.862348 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c547c6-30fd-43be-ac27-299da9065a25-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.862657 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.862637 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.862763 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.862719 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbm26\" (UniqueName: \"kubernetes.io/projected/54c547c6-30fd-43be-ac27-299da9065a25-kube-api-access-rbm26\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.862763 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.862755 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c547c6-30fd-43be-ac27-299da9065a25-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.862877 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.862797 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.862877 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.862856 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.862990 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.862946 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.863056 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.862984 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c547c6-30fd-43be-ac27-299da9065a25-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.863056 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.863026 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.863159 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.863120 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.863690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.863394 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.863690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.863629 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.864011 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.863985 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c547c6-30fd-43be-ac27-299da9065a25-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.864344 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.864322 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.866570 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.866545 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c547c6-30fd-43be-ac27-299da9065a25-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.866987 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.866963 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.931695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.930893 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c547c6-30fd-43be-ac27-299da9065a25-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:38.931695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:38.931164 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbm26\" (UniqueName: \"kubernetes.io/projected/54c547c6-30fd-43be-ac27-299da9065a25-kube-api-access-rbm26\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:39.216329 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.216205 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:39.261089 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.261058 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74"] Nov 02 01:43:39.266332 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.266309 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.296322 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.296174 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:43:39.331370 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.331327 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74"] Nov 02 01:43:39.369578 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.369490 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.369769 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.369666 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.369769 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.369728 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zgk4\" (UniqueName: \"kubernetes.io/projected/5952e0fe-3c89-4ec9-b8b0-79842c388b67-kube-api-access-6zgk4\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.369907 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.369774 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.369907 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.369827 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.370360 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.369882 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.370472 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.370454 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.370556 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.370540 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.370620 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.370575 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.395413 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.395381 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" event={"ID":"2320a74e-3e24-4633-8d10-807bcafda325","Type":"ContainerStarted","Data":"6b7fd1c534894d74d48fa13febedbec7b82d5d6de4e03749b4bdb806adb6f038"} Nov 02 01:43:39.395550 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.395444 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" event={"ID":"2320a74e-3e24-4633-8d10-807bcafda325","Type":"ContainerStarted","Data":"1c7d89690bcb1ddc2028f4213fba253f03e30c33f81d039def8a37f50a441d13"} Nov 02 01:43:39.471521 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.471454 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.471521 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.471485 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zgk4\" (UniqueName: \"kubernetes.io/projected/5952e0fe-3c89-4ec9-b8b0-79842c388b67-kube-api-access-6zgk4\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.471521 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.471509 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.471801 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.471738 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.471801 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.471783 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.471913 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.471822 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.471913 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.471902 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.472022 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.471931 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.472022 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.471967 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.472131 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.472088 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-credential-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.472185 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.472131 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-workload-certs\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.472375 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.472353 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-data\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.472563 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.472541 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istiod-ca-cert\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.472645 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.472621 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-workload-socket\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.474351 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.474328 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-envoy\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.474564 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.474547 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-podinfo\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.528118 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.528048 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" podStartSLOduration=2.52803169 podStartE2EDuration="2.52803169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:39.521307189 +0000 UTC m=+1136.192559871" watchObservedRunningTime="2026-11-02 01:43:39.52803169 +0000 UTC m=+1136.199284435" Nov 02 01:43:39.544691 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.542487 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zgk4\" (UniqueName: \"kubernetes.io/projected/5952e0fe-3c89-4ec9-b8b0-79842c388b67-kube-api-access-6zgk4\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.545706 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.545624 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-token\") pod \"e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.597162 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.597123 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:39.705774 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.705748 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt"] Nov 02 01:43:39.710365 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.709524 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:43:39.710365 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.709603 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:43:39.710365 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.709646 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:43:39.993594 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.993558 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74"] Nov 02 01:43:39.994940 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:43:39.994904 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5952e0fe_3c89_4ec9_b8b0_79842c388b67.slice/crio-8b47e6a8fd5dd9b988d13568441f779e970e4ca57c61ad75bc99ceed2ac062dd WatchSource:0}: Error finding container 8b47e6a8fd5dd9b988d13568441f779e970e4ca57c61ad75bc99ceed2ac062dd: Status 404 returned error can't find the container with id 8b47e6a8fd5dd9b988d13568441f779e970e4ca57c61ad75bc99ceed2ac062dd Nov 02 01:43:39.999369 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.999334 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:43:39.999477 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.999416 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:43:39.999543 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:39.999479 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:43:40.203811 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:40.202230 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:40.203811 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:40.202419 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 01:43:40.203811 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:40.202466 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" podUID="2320a74e-3e24-4633-8d10-807bcafda325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 01:43:40.425698 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:40.424328 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" event={"ID":"54c547c6-30fd-43be-ac27-299da9065a25","Type":"ContainerStarted","Data":"9dac6e5a3e1ed2c1fe6d79d550399d27f0b1f199cb175e1645552506c7f32ad9"} Nov 02 01:43:40.425698 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:40.424412 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" event={"ID":"54c547c6-30fd-43be-ac27-299da9065a25","Type":"ContainerStarted","Data":"8a01b8a79550fe724e3e05646cef68d71f65e7a4ee6b9733142ca010b5324465"} Nov 02 01:43:40.429753 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:40.427873 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" event={"ID":"5952e0fe-3c89-4ec9-b8b0-79842c388b67","Type":"ContainerStarted","Data":"c70cba7777ad36b84fd21d150ee3dc9d823708a6fc55e180ed645028cca3e976"} Nov 02 01:43:40.429753 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:40.427927 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" event={"ID":"5952e0fe-3c89-4ec9-b8b0-79842c388b67","Type":"ContainerStarted","Data":"8b47e6a8fd5dd9b988d13568441f779e970e4ca57c61ad75bc99ceed2ac062dd"} Nov 02 01:43:40.549904 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:40.549841 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" podStartSLOduration=2.549819939 podStartE2EDuration="2.549819939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:40.547874144 +0000 UTC m=+1137.219126832" watchObservedRunningTime="2026-11-02 01:43:40.549819939 +0000 UTC m=+1137.221072618" Nov 02 01:43:40.605697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:40.599766 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:40.683751 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:40.682155 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" podStartSLOduration=1.682134818 podStartE2EDuration="1.682134818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:43:40.681610226 +0000 UTC m=+1137.352862907" watchObservedRunningTime="2026-11-02 01:43:40.682134818 +0000 UTC m=+1137.353387504" Nov 02 01:43:41.212692 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:41.206098 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 01:43:41.212692 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:41.206167 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" podUID="2320a74e-3e24-4633-8d10-807bcafda325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 01:43:41.237698 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:41.216880 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:41.237698 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:41.218986 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 01:43:41.237698 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:41.219040 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" podUID="54c547c6-30fd-43be-ac27-299da9065a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 01:43:41.602693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:41.602007 5044 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 02 01:43:41.602693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:41.602073 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" podUID="5952e0fe-3c89-4ec9-b8b0-79842c388b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 02 01:43:42.031805 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:42.001772 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:43:42.108722 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:42.108653 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:43:42.243715 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:42.230138 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 01:43:42.243715 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:42.230197 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" podUID="54c547c6-30fd-43be-ac27-299da9065a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 01:43:42.292770 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:42.290521 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 01:43:42.292770 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:42.290579 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" podUID="2320a74e-3e24-4633-8d10-807bcafda325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 01:43:42.315811 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:42.315745 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:43:42.317277 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:42.317229 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:43:42.382846 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:42.382795 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:43:42.448156 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:42.448116 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:43:42.547924 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:42.547758 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:43:42.566811 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:42.566754 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:43:42.602158 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:42.602124 5044 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 02 01:43:42.602464 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:42.602419 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" podUID="5952e0fe-3c89-4ec9-b8b0-79842c388b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 02 01:43:42.886627 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:42.885871 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 01:43:43.202629 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:43.202114 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 01:43:43.202629 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:43.202177 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" podUID="2320a74e-3e24-4633-8d10-807bcafda325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 01:43:43.219209 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:43.219081 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 01:43:43.219209 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:43.219141 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" podUID="54c547c6-30fd-43be-ac27-299da9065a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 01:43:43.601695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:43.600767 5044 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 02 01:43:43.601695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:43.600825 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" podUID="5952e0fe-3c89-4ec9-b8b0-79842c388b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 02 01:43:44.217695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:44.208082 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 01:43:44.217695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:44.208140 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" podUID="2320a74e-3e24-4633-8d10-807bcafda325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 01:43:44.217695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:44.217003 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 01:43:44.217695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:44.217054 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" podUID="54c547c6-30fd-43be-ac27-299da9065a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 01:43:44.598697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:44.598043 5044 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 02 01:43:44.598697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:44.598105 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" podUID="5952e0fe-3c89-4ec9-b8b0-79842c388b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 02 01:43:45.201690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:45.201416 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 01:43:45.201690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:45.201477 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" podUID="2320a74e-3e24-4633-8d10-807bcafda325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 01:43:45.220704 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:45.220011 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 01:43:45.220704 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:45.220073 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" podUID="54c547c6-30fd-43be-ac27-299da9065a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 01:43:45.646494 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:45.644017 5044 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 02 01:43:45.646494 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:45.644074 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" podUID="5952e0fe-3c89-4ec9-b8b0-79842c388b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 02 01:43:46.206697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:46.203352 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 02 01:43:46.206697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:46.203420 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" podUID="2320a74e-3e24-4633-8d10-807bcafda325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 02 01:43:46.225696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:46.221077 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 02 01:43:46.225696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:46.221138 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" podUID="54c547c6-30fd-43be-ac27-299da9065a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 02 01:43:46.598059 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:46.598021 5044 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 02 01:43:46.598344 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:46.598312 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" podUID="5952e0fe-3c89-4ec9-b8b0-79842c388b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 02 01:43:47.392199 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:47.392149 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:43:47.947357 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:47.947315 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:47.948357 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:47.948331 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:47.949919 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:47.949884 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 01:43:48.191764 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:48.191733 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:48.192717 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:48.192693 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:48.279690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:48.279106 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 01:43:48.535039 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:48.534960 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:48.536356 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:48.536323 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:48.538871 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:48.538848 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:43:52.171160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:52.171124 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:43:52.323168 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:52.323132 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:43:54.172225 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:43:54.172187 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:43:56.916193 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:43:56.916155 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:00.044293 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:00.044258 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:01.701634 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:01.701592 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:06.481636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:06.481599 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:11.275407 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:11.275373 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:12.332511 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:12.332475 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr"] Nov 02 01:44:12.333026 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:12.332768 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" podUID="d13c6464-cb62-45eb-8cad-ca76333f3811" containerName="istio-proxy" containerID="cri-o://b8f61f11170ec304638a8e72f20f5519eaec6284a4e619c9060a3bab895b9a93" gracePeriod=30 Nov 02 01:44:13.517846 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:13.517812 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:16.080123 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:16.080077 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:17.596367 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.596301 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:44:17.692104 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.692076 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-token\" (UniqueName: \"kubernetes.io/projected/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-token\") pod \"d13c6464-cb62-45eb-8cad-ca76333f3811\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " Nov 02 01:44:17.692230 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.692114 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-podinfo\") pod \"d13c6464-cb62-45eb-8cad-ca76333f3811\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " Nov 02 01:44:17.692230 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.692140 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-credential-socket\") pod \"d13c6464-cb62-45eb-8cad-ca76333f3811\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " Nov 02 01:44:17.692341 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.692320 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-data\") pod \"d13c6464-cb62-45eb-8cad-ca76333f3811\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " Nov 02 01:44:17.692400 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.692370 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d13c6464-cb62-45eb-8cad-ca76333f3811-kube-api-access-fltrt\" (UniqueName: \"kubernetes.io/projected/d13c6464-cb62-45eb-8cad-ca76333f3811-kube-api-access-fltrt\") pod \"d13c6464-cb62-45eb-8cad-ca76333f3811\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " Nov 02 01:44:17.692495 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.692406 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-credential-socket" pod "d13c6464-cb62-45eb-8cad-ca76333f3811" (UID: "d13c6464-cb62-45eb-8cad-ca76333f3811"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:17.692495 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.692410 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-workload-socket\") pod \"d13c6464-cb62-45eb-8cad-ca76333f3811\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " Nov 02 01:44:17.692601 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.692521 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-envoy\") pod \"d13c6464-cb62-45eb-8cad-ca76333f3811\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " Nov 02 01:44:17.692601 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.692551 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-data" pod "d13c6464-cb62-45eb-8cad-ca76333f3811" (UID: "d13c6464-cb62-45eb-8cad-ca76333f3811"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:17.692601 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.692572 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-workload-socket" pod "d13c6464-cb62-45eb-8cad-ca76333f3811" (UID: "d13c6464-cb62-45eb-8cad-ca76333f3811"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:17.692790 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.692699 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-workload-certs\") pod \"d13c6464-cb62-45eb-8cad-ca76333f3811\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " Nov 02 01:44:17.692790 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.692729 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d13c6464-cb62-45eb-8cad-ca76333f3811-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d13c6464-cb62-45eb-8cad-ca76333f3811-istiod-ca-cert\") pod \"d13c6464-cb62-45eb-8cad-ca76333f3811\" (UID: \"d13c6464-cb62-45eb-8cad-ca76333f3811\") " Nov 02 01:44:17.692991 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.692958 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-workload-certs" pod "d13c6464-cb62-45eb-8cad-ca76333f3811" (UID: "d13c6464-cb62-45eb-8cad-ca76333f3811"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:17.693274 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.693252 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13c6464-cb62-45eb-8cad-ca76333f3811-istiod-ca-cert" pod "d13c6464-cb62-45eb-8cad-ca76333f3811" (UID: "d13c6464-cb62-45eb-8cad-ca76333f3811"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:44:17.693365 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.693254 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:44:17.693365 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.693301 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:44:17.693365 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.693320 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:44:17.693365 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.693335 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:44:17.694476 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.694459 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-token" pod "d13c6464-cb62-45eb-8cad-ca76333f3811" (UID: "d13c6464-cb62-45eb-8cad-ca76333f3811"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:17.694811 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.694791 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-podinfo" pod "d13c6464-cb62-45eb-8cad-ca76333f3811" (UID: "d13c6464-cb62-45eb-8cad-ca76333f3811"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:44:17.694872 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.694804 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13c6464-cb62-45eb-8cad-ca76333f3811-kube-api-access-fltrt" pod "d13c6464-cb62-45eb-8cad-ca76333f3811" (UID: "d13c6464-cb62-45eb-8cad-ca76333f3811"). InnerVolumeSpecName "kube-api-access-fltrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:44:17.694909 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.694873 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-envoy" pod "d13c6464-cb62-45eb-8cad-ca76333f3811" (UID: "d13c6464-cb62-45eb-8cad-ca76333f3811"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:44:17.794525 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.794494 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:44:17.794525 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.794518 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:44:17.794525 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.794529 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fltrt\" (UniqueName: \"kubernetes.io/projected/d13c6464-cb62-45eb-8cad-ca76333f3811-kube-api-access-fltrt\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:44:17.794827 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.794539 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d13c6464-cb62-45eb-8cad-ca76333f3811-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:44:17.794827 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.794547 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d13c6464-cb62-45eb-8cad-ca76333f3811-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:44:17.854936 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.854851 5044 generic.go:356] "Generic (PLEG): container finished" podID="d13c6464-cb62-45eb-8cad-ca76333f3811" containerID="b8f61f11170ec304638a8e72f20f5519eaec6284a4e619c9060a3bab895b9a93" exitCode=0 Nov 02 01:44:17.854936 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.854922 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" Nov 02 01:44:17.855105 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.854948 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" event={"ID":"d13c6464-cb62-45eb-8cad-ca76333f3811","Type":"ContainerDied","Data":"b8f61f11170ec304638a8e72f20f5519eaec6284a4e619c9060a3bab895b9a93"} Nov 02 01:44:17.855105 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.855042 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr" event={"ID":"d13c6464-cb62-45eb-8cad-ca76333f3811","Type":"ContainerDied","Data":"c54882d1c55844bdb540bbb9a96d75d43c79a0cb50bbe9b59311b87ff99ddb13"} Nov 02 01:44:17.855105 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.855072 5044 scope.go:122] "RemoveContainer" containerID="b8f61f11170ec304638a8e72f20f5519eaec6284a4e619c9060a3bab895b9a93" Nov 02 01:44:17.865450 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.865432 5044 scope.go:122] "RemoveContainer" containerID="b8f61f11170ec304638a8e72f20f5519eaec6284a4e619c9060a3bab895b9a93" Nov 02 01:44:17.865780 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:44:17.865757 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f61f11170ec304638a8e72f20f5519eaec6284a4e619c9060a3bab895b9a93\": container with ID starting with b8f61f11170ec304638a8e72f20f5519eaec6284a4e619c9060a3bab895b9a93 not found: ID does not exist" containerID="b8f61f11170ec304638a8e72f20f5519eaec6284a4e619c9060a3bab895b9a93" Nov 02 01:44:17.865842 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:17.865790 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f61f11170ec304638a8e72f20f5519eaec6284a4e619c9060a3bab895b9a93"} err="failed to get container status \"b8f61f11170ec304638a8e72f20f5519eaec6284a4e619c9060a3bab895b9a93\": rpc error: code = NotFound desc = could not find container \"b8f61f11170ec304638a8e72f20f5519eaec6284a4e619c9060a3bab895b9a93\": container with ID starting with b8f61f11170ec304638a8e72f20f5519eaec6284a4e619c9060a3bab895b9a93 not found: ID does not exist" Nov 02 01:44:18.024435 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:18.024403 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr"] Nov 02 01:44:18.028788 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:18.028763 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2qlpr"] Nov 02 01:44:19.453054 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:19.453019 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d13c6464-cb62-45eb-8cad-ca76333f3811" path="/var/lib/kubelet/pods/d13c6464-cb62-45eb-8cad-ca76333f3811/volumes" Nov 02 01:44:20.876571 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:20.876535 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:23.668523 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.668482 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f"] Nov 02 01:44:23.669069 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.669041 5044 cpu_manager.go:414] "containerMap: removing container" podUID="d13c6464-cb62-45eb-8cad-ca76333f3811" containerName="istio-proxy" Nov 02 01:44:23.669069 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.669053 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13c6464-cb62-45eb-8cad-ca76333f3811" containerName="istio-proxy" Nov 02 01:44:23.669188 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.669153 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13c6464-cb62-45eb-8cad-ca76333f3811" containerName="istio-proxy" Nov 02 01:44:23.671990 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.671969 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.760784 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.760737 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.760965 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.760815 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.760965 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.760923 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.761072 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.761002 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.761072 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.761037 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.761175 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.761108 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.761175 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.761141 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.761251 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.761218 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph7d9\" (UniqueName: \"kubernetes.io/projected/98be750b-4e74-4378-871b-3a3e9d53cd78-kube-api-access-ph7d9\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.761301 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.761282 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98be750b-4e74-4378-871b-3a3e9d53cd78-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.769783 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.767829 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f"] Nov 02 01:44:23.861973 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.861942 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.862177 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.861993 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.862177 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.862026 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.862177 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.862058 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.862177 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.862085 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.862177 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.862130 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.862177 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.862157 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.862395 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.862201 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph7d9\" (UniqueName: \"kubernetes.io/projected/98be750b-4e74-4378-871b-3a3e9d53cd78-kube-api-access-ph7d9\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.862395 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.862245 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98be750b-4e74-4378-871b-3a3e9d53cd78-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.862460 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.862424 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.862851 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.862827 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.862986 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.862906 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98be750b-4e74-4378-871b-3a3e9d53cd78-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.863095 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.863078 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.863265 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.863251 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.865042 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.865015 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.865359 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.865336 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.924841 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.924757 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.924980 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.924900 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph7d9\" (UniqueName: \"kubernetes.io/projected/98be750b-4e74-4378-871b-3a3e9d53cd78-kube-api-access-ph7d9\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:23.987802 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:23.987767 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:24.174338 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:44:24.174292 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:44:24.427758 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:24.427732 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f"] Nov 02 01:44:24.428820 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:44:24.428794 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98be750b_4e74_4378_871b_3a3e9d53cd78.slice/crio-92348f2eee7c3343f10cea8c5778625b737536301eadce8f439cb5ef0449ca59 WatchSource:0}: Error finding container 92348f2eee7c3343f10cea8c5778625b737536301eadce8f439cb5ef0449ca59: Status 404 returned error can't find the container with id 92348f2eee7c3343f10cea8c5778625b737536301eadce8f439cb5ef0449ca59 Nov 02 01:44:24.431212 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:24.431182 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:44:24.431285 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:24.431246 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:44:24.431285 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:24.431279 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:44:24.901925 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:24.901888 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" event={"ID":"98be750b-4e74-4378-871b-3a3e9d53cd78","Type":"ContainerStarted","Data":"f49c280fc18c14ccfa7c4c3c659dabc621db819e2a3200589db9c35475e37122"} Nov 02 01:44:24.902409 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:24.901982 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" event={"ID":"98be750b-4e74-4378-871b-3a3e9d53cd78","Type":"ContainerStarted","Data":"92348f2eee7c3343f10cea8c5778625b737536301eadce8f439cb5ef0449ca59"} Nov 02 01:44:24.988876 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:24.988848 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:25.025731 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:25.025684 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" podStartSLOduration=2.025658014 podStartE2EDuration="2.025658014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:25.023185427 +0000 UTC m=+1181.694438108" watchObservedRunningTime="2026-11-02 01:44:25.025658014 +0000 UTC m=+1181.696910744" Nov 02 01:44:25.690695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:25.690306 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:25.991754 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:25.991486 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 01:44:25.991754 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:25.991561 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" podUID="98be750b-4e74-4378-871b-3a3e9d53cd78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 01:44:26.792763 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:26.792723 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b"] Nov 02 01:44:26.796634 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:26.796608 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:26.846445 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:26.846410 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b"] Nov 02 01:44:26.897694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:26.897272 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-187478\"/\"distinct-w4-187478-kserve-self-signed-certs\"" Nov 02 01:44:26.898496 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:26.898418 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:26.911433 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:26.911389 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70609eac-cd4a-4e96-84b0-536dccf3a715-tls-certs\") pod \"distinct-w4-187478-kserve-76df86f488-mrd5b\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:26.911597 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:26.911457 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-home\") pod \"distinct-w4-187478-kserve-76df86f488-mrd5b\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:26.911597 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:26.911514 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-model-cache\") pod \"distinct-w4-187478-kserve-76df86f488-mrd5b\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:26.911735 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:26.911691 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-dshm\") pod \"distinct-w4-187478-kserve-76df86f488-mrd5b\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:26.911792 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:26.911770 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-tmp-dir\") pod \"distinct-w4-187478-kserve-76df86f488-mrd5b\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:26.911892 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:26.911866 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hpv4\" (UniqueName: \"kubernetes.io/projected/70609eac-cd4a-4e96-84b0-536dccf3a715-kube-api-access-2hpv4\") pod \"distinct-w4-187478-kserve-76df86f488-mrd5b\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:26.998700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:26.997072 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 01:44:26.998700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:26.997136 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" podUID="98be750b-4e74-4378-871b-3a3e9d53cd78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 01:44:27.013532 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:27.013492 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-home\") pod \"distinct-w4-187478-kserve-76df86f488-mrd5b\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:27.013728 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:27.013557 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-model-cache\") pod \"distinct-w4-187478-kserve-76df86f488-mrd5b\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:27.013728 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:27.013651 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-dshm\") pod \"distinct-w4-187478-kserve-76df86f488-mrd5b\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:27.013887 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:27.013731 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-tmp-dir\") pod \"distinct-w4-187478-kserve-76df86f488-mrd5b\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:27.013887 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:27.013757 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hpv4\" (UniqueName: \"kubernetes.io/projected/70609eac-cd4a-4e96-84b0-536dccf3a715-kube-api-access-2hpv4\") pod \"distinct-w4-187478-kserve-76df86f488-mrd5b\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:27.013887 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:27.013804 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70609eac-cd4a-4e96-84b0-536dccf3a715-tls-certs\") pod \"distinct-w4-187478-kserve-76df86f488-mrd5b\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:27.014072 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:27.014044 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-home\") pod \"distinct-w4-187478-kserve-76df86f488-mrd5b\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:27.014186 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:27.014160 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-model-cache\") pod \"distinct-w4-187478-kserve-76df86f488-mrd5b\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:27.014256 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:27.014207 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-tmp-dir\") pod \"distinct-w4-187478-kserve-76df86f488-mrd5b\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:27.016436 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:27.016412 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-dshm\") pod \"distinct-w4-187478-kserve-76df86f488-mrd5b\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:27.016591 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:27.016572 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70609eac-cd4a-4e96-84b0-536dccf3a715-tls-certs\") pod \"distinct-w4-187478-kserve-76df86f488-mrd5b\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:27.088728 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:27.088125 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hpv4\" (UniqueName: \"kubernetes.io/projected/70609eac-cd4a-4e96-84b0-536dccf3a715-kube-api-access-2hpv4\") pod \"distinct-w4-187478-kserve-76df86f488-mrd5b\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:27.115093 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:27.115063 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:27.543238 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:27.543200 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b"] Nov 02 01:44:27.544441 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:44:27.544411 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70609eac_cd4a_4e96_84b0_536dccf3a715.slice/crio-eac4152d5c8ef23fe776dfe66bbb6dc998d8af3b9a0ccf1c159a3674d7a3f53c WatchSource:0}: Error finding container eac4152d5c8ef23fe776dfe66bbb6dc998d8af3b9a0ccf1c159a3674d7a3f53c: Status 404 returned error can't find the container with id eac4152d5c8ef23fe776dfe66bbb6dc998d8af3b9a0ccf1c159a3674d7a3f53c Nov 02 01:44:27.929922 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:27.929821 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" event={"ID":"70609eac-cd4a-4e96-84b0-536dccf3a715","Type":"ContainerStarted","Data":"2a51530d5b80a2dc8a3768ce449c36e826b71766b01f37c849ec9258991bfc3c"} Nov 02 01:44:27.929922 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:27.929876 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" event={"ID":"70609eac-cd4a-4e96-84b0-536dccf3a715","Type":"ContainerStarted","Data":"eac4152d5c8ef23fe776dfe66bbb6dc998d8af3b9a0ccf1c159a3674d7a3f53c"} Nov 02 01:44:27.992155 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:27.991791 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 01:44:27.992155 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:27.991858 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" podUID="98be750b-4e74-4378-871b-3a3e9d53cd78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 01:44:28.057240 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:28.057187 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podStartSLOduration=2.057173214 podStartE2EDuration="2.057173214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:28.055497234 +0000 UTC m=+1184.726749917" watchObservedRunningTime="2026-11-02 01:44:28.057173214 +0000 UTC m=+1184.728425938" Nov 02 01:44:28.115468 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:28.115435 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:28.989095 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:28.989058 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 01:44:28.989304 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:28.989125 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" podUID="98be750b-4e74-4378-871b-3a3e9d53cd78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 01:44:29.992693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:29.992051 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 01:44:29.992693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:29.992112 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" podUID="98be750b-4e74-4378-871b-3a3e9d53cd78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 01:44:30.913871 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:30.913833 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:30.989877 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:30.989844 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 02 01:44:30.990083 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:30.989895 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" podUID="98be750b-4e74-4378-871b-3a3e9d53cd78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 02 01:44:31.993205 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:31.993176 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:31.993639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:31.993622 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:31.994374 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:31.994354 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:44:35.694514 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:35.694480 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:37.115576 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:37.115518 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:37.128376 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:37.128345 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:37.530123 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:37.530085 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:37.673796 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:37.673762 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:38.011169 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:38.011147 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:44:38.135913 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:38.135883 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:39.630662 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:39.630628 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:41.109198 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:41.109156 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:43.571866 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:43.571835 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 01:44:43.575872 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:43.575852 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 01:44:45.272501 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:45.272470 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:50.065847 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:50.065809 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:52.278590 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.278557 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h"] Nov 02 01:44:52.284111 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.284089 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:52.311554 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.311522 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-model-cache\") pod \"distinct-w0-b71ba5-kserve-86946c569f-z469h\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:52.311730 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.311565 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-dshm\") pod \"distinct-w0-b71ba5-kserve-86946c569f-z469h\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:52.311730 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.311663 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-tls-certs\") pod \"distinct-w0-b71ba5-kserve-86946c569f-z469h\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:52.311854 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.311742 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpk76\" (UniqueName: \"kubernetes.io/projected/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-kube-api-access-cpk76\") pod \"distinct-w0-b71ba5-kserve-86946c569f-z469h\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:52.311854 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.311808 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-home\") pod \"distinct-w0-b71ba5-kserve-86946c569f-z469h\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:52.311968 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.311914 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-tmp-dir\") pod \"distinct-w0-b71ba5-kserve-86946c569f-z469h\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:52.341318 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.341287 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h"] Nov 02 01:44:52.377458 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.377422 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:52.387460 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.387438 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b71ba5\"/\"distinct-w0-b71ba5-kserve-self-signed-certs\"" Nov 02 01:44:52.412721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.412696 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-tls-certs\") pod \"distinct-w0-b71ba5-kserve-86946c569f-z469h\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:52.412871 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.412759 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpk76\" (UniqueName: \"kubernetes.io/projected/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-kube-api-access-cpk76\") pod \"distinct-w0-b71ba5-kserve-86946c569f-z469h\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:52.412871 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.412838 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-home\") pod \"distinct-w0-b71ba5-kserve-86946c569f-z469h\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:52.412989 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.412971 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-tmp-dir\") pod \"distinct-w0-b71ba5-kserve-86946c569f-z469h\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:52.413067 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.413051 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-model-cache\") pod \"distinct-w0-b71ba5-kserve-86946c569f-z469h\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:52.413127 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.413081 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-dshm\") pod \"distinct-w0-b71ba5-kserve-86946c569f-z469h\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:52.413328 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.413304 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-home\") pod \"distinct-w0-b71ba5-kserve-86946c569f-z469h\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:52.413445 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.413312 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-tmp-dir\") pod \"distinct-w0-b71ba5-kserve-86946c569f-z469h\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:52.413445 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.413403 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-model-cache\") pod \"distinct-w0-b71ba5-kserve-86946c569f-z469h\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:52.415297 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.415266 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-dshm\") pod \"distinct-w0-b71ba5-kserve-86946c569f-z469h\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:52.415398 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.415316 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-tls-certs\") pod \"distinct-w0-b71ba5-kserve-86946c569f-z469h\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:52.475312 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.475279 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpk76\" (UniqueName: \"kubernetes.io/projected/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-kube-api-access-cpk76\") pod \"distinct-w0-b71ba5-kserve-86946c569f-z469h\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:52.598771 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.598663 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:52.994414 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:52.994385 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h"] Nov 02 01:44:52.996023 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:44:52.995984 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ad28d87_78de_4b69_9f2a_4d713cbac5ad.slice/crio-1bdcc4e0819773369eb6ebde099ca01c932744dd9b5306b6aac68e7d72ddf00e WatchSource:0}: Error finding container 1bdcc4e0819773369eb6ebde099ca01c932744dd9b5306b6aac68e7d72ddf00e: Status 404 returned error can't find the container with id 1bdcc4e0819773369eb6ebde099ca01c932744dd9b5306b6aac68e7d72ddf00e Nov 02 01:44:53.092002 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:53.091969 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" event={"ID":"7ad28d87-78de-4b69-9f2a-4d713cbac5ad","Type":"ContainerStarted","Data":"4d053eb250622dd6af25bc2eacebd4fd9f0b6fb873da60b1d2750a1318e3db10"} Nov 02 01:44:53.092133 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:53.092022 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" event={"ID":"7ad28d87-78de-4b69-9f2a-4d713cbac5ad","Type":"ContainerStarted","Data":"1bdcc4e0819773369eb6ebde099ca01c932744dd9b5306b6aac68e7d72ddf00e"} Nov 02 01:44:53.213069 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:53.213015 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podStartSLOduration=1.213000636 podStartE2EDuration="1.213000636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:44:53.210624618 +0000 UTC m=+1209.881877300" watchObservedRunningTime="2026-11-02 01:44:53.213000636 +0000 UTC m=+1209.884253343" Nov 02 01:44:53.599541 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:53.599509 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:44:54.174339 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:44:54.174295 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:44:54.882145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:54.882109 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:44:59.669590 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:44:59.669557 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:45:00.175354 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.175323 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5"] Nov 02 01:45:00.179230 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.179214 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5" Nov 02 01:45:00.192323 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.192300 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b"] Nov 02 01:45:00.195940 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.195923 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq"] Nov 02 01:45:00.196060 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.196042 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b" Nov 02 01:45:00.199429 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.199412 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq" Nov 02 01:45:00.284274 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.284239 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5"] Nov 02 01:45:00.311491 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.311460 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea76657a-20d2-4c01-923a-422f382b53d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5\" (UID: \"ea76657a-20d2-4c01-923a-422f382b53d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5" Nov 02 01:45:00.311638 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.311507 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2ll9\" (UniqueName: \"kubernetes.io/projected/d71aea2c-b510-44e6-98b8-35a4676028b1-kube-api-access-k2ll9\") pod \"maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b\" (UID: \"d71aea2c-b510-44e6-98b8-35a4676028b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b" Nov 02 01:45:00.311638 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.311601 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq727\" (UniqueName: \"kubernetes.io/projected/d83028cc-a184-419e-b378-eb0d15e7c976-kube-api-access-cq727\") pod \"maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq\" (UID: \"d83028cc-a184-419e-b378-eb0d15e7c976\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq" Nov 02 01:45:00.311789 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.311747 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4svt9\" (UniqueName: \"kubernetes.io/projected/ea76657a-20d2-4c01-923a-422f382b53d8-kube-api-access-4svt9\") pod \"maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5\" (UID: \"ea76657a-20d2-4c01-923a-422f382b53d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5" Nov 02 01:45:00.311962 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.311936 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71aea2c-b510-44e6-98b8-35a4676028b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b\" (UID: \"d71aea2c-b510-44e6-98b8-35a4676028b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b" Nov 02 01:45:00.312011 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.311977 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d83028cc-a184-419e-b378-eb0d15e7c976-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq\" (UID: \"d83028cc-a184-419e-b378-eb0d15e7c976\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq" Nov 02 01:45:00.323771 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.323747 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq"] Nov 02 01:45:00.330572 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.330547 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b"] Nov 02 01:45:00.336004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.335981 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\"" Nov 02 01:45:00.413689 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.413643 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71aea2c-b510-44e6-98b8-35a4676028b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b\" (UID: \"d71aea2c-b510-44e6-98b8-35a4676028b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b" Nov 02 01:45:00.413801 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.413708 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d83028cc-a184-419e-b378-eb0d15e7c976-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq\" (UID: \"d83028cc-a184-419e-b378-eb0d15e7c976\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq" Nov 02 01:45:00.413801 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.413775 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea76657a-20d2-4c01-923a-422f382b53d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5\" (UID: \"ea76657a-20d2-4c01-923a-422f382b53d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5" Nov 02 01:45:00.413911 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.413802 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2ll9\" (UniqueName: \"kubernetes.io/projected/d71aea2c-b510-44e6-98b8-35a4676028b1-kube-api-access-k2ll9\") pod \"maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b\" (UID: \"d71aea2c-b510-44e6-98b8-35a4676028b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b" Nov 02 01:45:00.413911 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.413821 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq727\" (UniqueName: \"kubernetes.io/projected/d83028cc-a184-419e-b378-eb0d15e7c976-kube-api-access-cq727\") pod \"maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq\" (UID: \"d83028cc-a184-419e-b378-eb0d15e7c976\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq" Nov 02 01:45:00.414016 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.413920 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4svt9\" (UniqueName: \"kubernetes.io/projected/ea76657a-20d2-4c01-923a-422f382b53d8-kube-api-access-4svt9\") pod \"maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5\" (UID: \"ea76657a-20d2-4c01-923a-422f382b53d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5" Nov 02 01:45:00.414460 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.414439 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d83028cc-a184-419e-b378-eb0d15e7c976-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq\" (UID: \"d83028cc-a184-419e-b378-eb0d15e7c976\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq" Nov 02 01:45:00.414547 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.414472 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71aea2c-b510-44e6-98b8-35a4676028b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b\" (UID: \"d71aea2c-b510-44e6-98b8-35a4676028b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b" Nov 02 01:45:00.414547 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.414487 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea76657a-20d2-4c01-923a-422f382b53d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5\" (UID: \"ea76657a-20d2-4c01-923a-422f382b53d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5" Nov 02 01:45:00.475584 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.475522 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4svt9\" (UniqueName: \"kubernetes.io/projected/ea76657a-20d2-4c01-923a-422f382b53d8-kube-api-access-4svt9\") pod \"maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5\" (UID: \"ea76657a-20d2-4c01-923a-422f382b53d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5" Nov 02 01:45:00.475584 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.475568 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2ll9\" (UniqueName: \"kubernetes.io/projected/d71aea2c-b510-44e6-98b8-35a4676028b1-kube-api-access-k2ll9\") pod \"maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b\" (UID: \"d71aea2c-b510-44e6-98b8-35a4676028b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b" Nov 02 01:45:00.475773 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.475686 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq727\" (UniqueName: \"kubernetes.io/projected/d83028cc-a184-419e-b378-eb0d15e7c976-kube-api-access-cq727\") pod \"maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq\" (UID: \"d83028cc-a184-419e-b378-eb0d15e7c976\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq" Nov 02 01:45:00.493727 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.493702 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5" Nov 02 01:45:00.509665 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.509638 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b" Nov 02 01:45:00.516260 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.516238 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq" Nov 02 01:45:00.892618 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:45:00.892589 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea76657a_20d2_4c01_923a_422f382b53d8.slice/crio-f53409a8862de3f052b7468cd6353322897e5cf68ad794f071408bea2b1fa2e7 WatchSource:0}: Error finding container f53409a8862de3f052b7468cd6353322897e5cf68ad794f071408bea2b1fa2e7: Status 404 returned error can't find the container with id f53409a8862de3f052b7468cd6353322897e5cf68ad794f071408bea2b1fa2e7 Nov 02 01:45:00.893032 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.892996 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5"] Nov 02 01:45:00.941443 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.941408 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq"] Nov 02 01:45:00.942212 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:45:00.942187 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83028cc_a184_419e_b378_eb0d15e7c976.slice/crio-f00e15b2cb3222823a482c4dde2f3ffc0711ce834bea4a89b0289c197fedb1d0 WatchSource:0}: Error finding container f00e15b2cb3222823a482c4dde2f3ffc0711ce834bea4a89b0289c197fedb1d0: Status 404 returned error can't find the container with id f00e15b2cb3222823a482c4dde2f3ffc0711ce834bea4a89b0289c197fedb1d0 Nov 02 01:45:00.945030 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:00.945009 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b"] Nov 02 01:45:00.945584 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:45:00.945566 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71aea2c_b510_44e6_98b8_35a4676028b1.slice/crio-757dfd45b843a8f286320873e21392e11c24793a3d77cf822f51706b734cf553 WatchSource:0}: Error finding container 757dfd45b843a8f286320873e21392e11c24793a3d77cf822f51706b734cf553: Status 404 returned error can't find the container with id 757dfd45b843a8f286320873e21392e11c24793a3d77cf822f51706b734cf553 Nov 02 01:45:01.153770 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:01.153692 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq" event={"ID":"d83028cc-a184-419e-b378-eb0d15e7c976","Type":"ContainerStarted","Data":"f00e15b2cb3222823a482c4dde2f3ffc0711ce834bea4a89b0289c197fedb1d0"} Nov 02 01:45:01.155021 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:01.154996 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b" event={"ID":"d71aea2c-b510-44e6-98b8-35a4676028b1","Type":"ContainerStarted","Data":"757dfd45b843a8f286320873e21392e11c24793a3d77cf822f51706b734cf553"} Nov 02 01:45:01.156277 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:01.156247 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5" event={"ID":"ea76657a-20d2-4c01-923a-422f382b53d8","Type":"ContainerStarted","Data":"f53409a8862de3f052b7468cd6353322897e5cf68ad794f071408bea2b1fa2e7"} Nov 02 01:45:02.165262 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:02.165174 5044 generic.go:356] "Generic (PLEG): container finished" podID="d83028cc-a184-419e-b378-eb0d15e7c976" containerID="a8102e4fcfdf95fdde186fab041fa3ba1772711c9a2e6bbcb228b0bd422725af" exitCode=60 Nov 02 01:45:02.165761 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:02.165261 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq" event={"ID":"d83028cc-a184-419e-b378-eb0d15e7c976","Type":"ContainerDied","Data":"a8102e4fcfdf95fdde186fab041fa3ba1772711c9a2e6bbcb228b0bd422725af"} Nov 02 01:45:02.165761 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:02.165507 5044 scope.go:122] "RemoveContainer" containerID="a8102e4fcfdf95fdde186fab041fa3ba1772711c9a2e6bbcb228b0bd422725af" Nov 02 01:45:02.166986 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:02.166963 5044 generic.go:356] "Generic (PLEG): container finished" podID="d71aea2c-b510-44e6-98b8-35a4676028b1" containerID="2e5a7bf454fbb70d73712d770e95d6d1ca9b11fd360d82d538149bf75890a40e" exitCode=60 Nov 02 01:45:02.167101 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:02.167071 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b" event={"ID":"d71aea2c-b510-44e6-98b8-35a4676028b1","Type":"ContainerDied","Data":"2e5a7bf454fbb70d73712d770e95d6d1ca9b11fd360d82d538149bf75890a40e"} Nov 02 01:45:02.167429 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:02.167409 5044 scope.go:122] "RemoveContainer" containerID="2e5a7bf454fbb70d73712d770e95d6d1ca9b11fd360d82d538149bf75890a40e" Nov 02 01:45:02.169409 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:02.169383 5044 generic.go:356] "Generic (PLEG): container finished" podID="ea76657a-20d2-4c01-923a-422f382b53d8" containerID="447c1d92aad7e17f596d225224152474c75d8b223abb2dd83f85c3b986e1e25a" exitCode=60 Nov 02 01:45:02.169522 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:02.169466 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5" event={"ID":"ea76657a-20d2-4c01-923a-422f382b53d8","Type":"ContainerDied","Data":"447c1d92aad7e17f596d225224152474c75d8b223abb2dd83f85c3b986e1e25a"} Nov 02 01:45:02.169725 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:02.169707 5044 scope.go:122] "RemoveContainer" containerID="447c1d92aad7e17f596d225224152474c75d8b223abb2dd83f85c3b986e1e25a" Nov 02 01:45:02.598903 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:02.598874 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:45:02.612883 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:02.612858 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:45:03.179288 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:03.179261 5044 generic.go:356] "Generic (PLEG): container finished" podID="d83028cc-a184-419e-b378-eb0d15e7c976" containerID="59df9824c6dfb70aeb38011d7a06af7a7cb5c982d7127c476a6610ae6c1fdc9f" exitCode=60 Nov 02 01:45:03.179688 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:03.179344 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq" event={"ID":"d83028cc-a184-419e-b378-eb0d15e7c976","Type":"ContainerDied","Data":"59df9824c6dfb70aeb38011d7a06af7a7cb5c982d7127c476a6610ae6c1fdc9f"} Nov 02 01:45:03.179688 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:03.179418 5044 scope.go:122] "RemoveContainer" containerID="a8102e4fcfdf95fdde186fab041fa3ba1772711c9a2e6bbcb228b0bd422725af" Nov 02 01:45:03.179790 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:03.179744 5044 scope.go:122] "RemoveContainer" containerID="59df9824c6dfb70aeb38011d7a06af7a7cb5c982d7127c476a6610ae6c1fdc9f" Nov 02 01:45:03.180119 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:45:03.180094 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq_odh-ai-gateway-infra(d83028cc-a184-419e-b378-eb0d15e7c976)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq" podUID="d83028cc-a184-419e-b378-eb0d15e7c976" Nov 02 01:45:03.186929 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:03.186902 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b" event={"ID":"d71aea2c-b510-44e6-98b8-35a4676028b1","Type":"ContainerStarted","Data":"4a45e14a2a418de77c2ae385b74b08407f0e4e1c2919cfe2de2c0b2e5b0220be"} Nov 02 01:45:03.187122 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:03.187105 5044 scope.go:122] "RemoveContainer" containerID="4a45e14a2a418de77c2ae385b74b08407f0e4e1c2919cfe2de2c0b2e5b0220be" Nov 02 01:45:03.187429 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:45:03.187407 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b_odh-ai-gateway-infra(d71aea2c-b510-44e6-98b8-35a4676028b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b" podUID="d71aea2c-b510-44e6-98b8-35a4676028b1" Nov 02 01:45:03.189309 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:03.189287 5044 generic.go:356] "Generic (PLEG): container finished" podID="ea76657a-20d2-4c01-923a-422f382b53d8" containerID="a97bb1cc9e9780d1bc714b24a9b4192c90fea678fad06711fb041d2225849e2d" exitCode=60 Nov 02 01:45:03.189411 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:03.189368 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5" event={"ID":"ea76657a-20d2-4c01-923a-422f382b53d8","Type":"ContainerDied","Data":"a97bb1cc9e9780d1bc714b24a9b4192c90fea678fad06711fb041d2225849e2d"} Nov 02 01:45:03.189591 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:03.189574 5044 scope.go:122] "RemoveContainer" containerID="a97bb1cc9e9780d1bc714b24a9b4192c90fea678fad06711fb041d2225849e2d" Nov 02 01:45:03.189844 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:45:03.189822 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5_odh-ai-gateway-infra(ea76657a-20d2-4c01-923a-422f382b53d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5" podUID="ea76657a-20d2-4c01-923a-422f382b53d8" Nov 02 01:45:03.201013 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:03.200992 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:45:03.317798 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:03.317775 5044 scope.go:122] "RemoveContainer" containerID="447c1d92aad7e17f596d225224152474c75d8b223abb2dd83f85c3b986e1e25a" Nov 02 01:45:04.198057 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:04.198029 5044 scope.go:122] "RemoveContainer" containerID="59df9824c6dfb70aeb38011d7a06af7a7cb5c982d7127c476a6610ae6c1fdc9f" Nov 02 01:45:04.198545 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:45:04.198354 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq_odh-ai-gateway-infra(d83028cc-a184-419e-b378-eb0d15e7c976)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq" podUID="d83028cc-a184-419e-b378-eb0d15e7c976" Nov 02 01:45:04.199765 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:04.199738 5044 generic.go:356] "Generic (PLEG): container finished" podID="d71aea2c-b510-44e6-98b8-35a4676028b1" containerID="4a45e14a2a418de77c2ae385b74b08407f0e4e1c2919cfe2de2c0b2e5b0220be" exitCode=60 Nov 02 01:45:04.199888 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:04.199821 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b" event={"ID":"d71aea2c-b510-44e6-98b8-35a4676028b1","Type":"ContainerDied","Data":"4a45e14a2a418de77c2ae385b74b08407f0e4e1c2919cfe2de2c0b2e5b0220be"} Nov 02 01:45:04.199941 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:04.199884 5044 scope.go:122] "RemoveContainer" containerID="2e5a7bf454fbb70d73712d770e95d6d1ca9b11fd360d82d538149bf75890a40e" Nov 02 01:45:04.200080 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:04.200062 5044 scope.go:122] "RemoveContainer" containerID="4a45e14a2a418de77c2ae385b74b08407f0e4e1c2919cfe2de2c0b2e5b0220be" Nov 02 01:45:04.200355 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:45:04.200333 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b_odh-ai-gateway-infra(d71aea2c-b510-44e6-98b8-35a4676028b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b" podUID="d71aea2c-b510-44e6-98b8-35a4676028b1" Nov 02 01:45:04.202869 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:04.202851 5044 scope.go:122] "RemoveContainer" containerID="a97bb1cc9e9780d1bc714b24a9b4192c90fea678fad06711fb041d2225849e2d" Nov 02 01:45:04.203091 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:45:04.203071 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5_odh-ai-gateway-infra(ea76657a-20d2-4c01-923a-422f382b53d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5" podUID="ea76657a-20d2-4c01-923a-422f382b53d8" Nov 02 01:45:04.476197 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:04.476116 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:45:05.210205 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:05.210180 5044 scope.go:122] "RemoveContainer" containerID="4a45e14a2a418de77c2ae385b74b08407f0e4e1c2919cfe2de2c0b2e5b0220be" Nov 02 01:45:05.210600 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:45:05.210466 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b_odh-ai-gateway-infra(d71aea2c-b510-44e6-98b8-35a4676028b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b" podUID="d71aea2c-b510-44e6-98b8-35a4676028b1" Nov 02 01:45:06.835066 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:06.835028 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-qwlzg-9pp4j"] Nov 02 01:45:06.840537 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:06.840518 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" Nov 02 01:45:06.900433 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:06.900387 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-qwlzg-9pp4j"] Nov 02 01:45:06.943435 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:06.943397 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pt8bk\"" Nov 02 01:45:06.997795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:06.997759 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-service-ca-bundle\") pod \"periodic-gathering-qwlzg-9pp4j\" (UID: \"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1\") " pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" Nov 02 01:45:06.997999 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:06.997896 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-archives-path\") pod \"periodic-gathering-qwlzg-9pp4j\" (UID: \"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1\") " pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" Nov 02 01:45:06.998064 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:06.998033 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh889\" (UniqueName: \"kubernetes.io/projected/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-kube-api-access-wh889\") pod \"periodic-gathering-qwlzg-9pp4j\" (UID: \"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1\") " pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" Nov 02 01:45:06.998064 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:06.998055 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-trusted-ca-bundle\") pod \"periodic-gathering-qwlzg-9pp4j\" (UID: \"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1\") " pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" Nov 02 01:45:07.099555 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:07.099465 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh889\" (UniqueName: \"kubernetes.io/projected/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-kube-api-access-wh889\") pod \"periodic-gathering-qwlzg-9pp4j\" (UID: \"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1\") " pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" Nov 02 01:45:07.099555 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:07.099505 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-trusted-ca-bundle\") pod \"periodic-gathering-qwlzg-9pp4j\" (UID: \"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1\") " pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" Nov 02 01:45:07.099800 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:07.099562 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-service-ca-bundle\") pod \"periodic-gathering-qwlzg-9pp4j\" (UID: \"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1\") " pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" Nov 02 01:45:07.099800 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:07.099630 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-archives-path\") pod \"periodic-gathering-qwlzg-9pp4j\" (UID: \"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1\") " pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" Nov 02 01:45:07.100155 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:07.100132 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-archives-path\") pod \"periodic-gathering-qwlzg-9pp4j\" (UID: \"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1\") " pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" Nov 02 01:45:07.100289 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:07.100269 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-service-ca-bundle\") pod \"periodic-gathering-qwlzg-9pp4j\" (UID: \"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1\") " pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" Nov 02 01:45:07.100537 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:07.100520 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-trusted-ca-bundle\") pod \"periodic-gathering-qwlzg-9pp4j\" (UID: \"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1\") " pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" Nov 02 01:45:07.162797 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:07.162762 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh889\" (UniqueName: \"kubernetes.io/projected/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-kube-api-access-wh889\") pod \"periodic-gathering-qwlzg-9pp4j\" (UID: \"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1\") " pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" Nov 02 01:45:07.457009 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:07.456928 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" Nov 02 01:45:07.842021 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:07.841994 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-qwlzg-9pp4j"] Nov 02 01:45:07.844479 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:45:07.844448 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0cdd1f8_c3bb_4387_8a14_c7b62b0641f1.slice/crio-e48040dc3a48dc8ca90588425d4478197fb655723cd4239f9eecf2c3b21f3c28 WatchSource:0}: Error finding container e48040dc3a48dc8ca90588425d4478197fb655723cd4239f9eecf2c3b21f3c28: Status 404 returned error can't find the container with id e48040dc3a48dc8ca90588425d4478197fb655723cd4239f9eecf2c3b21f3c28 Nov 02 01:45:08.229725 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:08.229604 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" event={"ID":"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1","Type":"ContainerStarted","Data":"eec6f1c6acd6ff69cb906b6fd10aed67859349f8de49ab55e10d800f277e4b9c"} Nov 02 01:45:08.229725 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:08.229655 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" event={"ID":"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1","Type":"ContainerStarted","Data":"e48040dc3a48dc8ca90588425d4478197fb655723cd4239f9eecf2c3b21f3c28"} Nov 02 01:45:08.348657 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:08.348602 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" podStartSLOduration=2.348588465 podStartE2EDuration="2.348588465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:08.346491922 +0000 UTC m=+1225.017744603" watchObservedRunningTime="2026-11-02 01:45:08.348588465 +0000 UTC m=+1225.019841147" Nov 02 01:45:08.532036 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:08.531983 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:45:09.295374 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:09.295321 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:45:14.072231 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:14.072193 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:45:14.445702 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:14.445592 5044 scope.go:122] "RemoveContainer" containerID="59df9824c6dfb70aeb38011d7a06af7a7cb5c982d7127c476a6610ae6c1fdc9f" Nov 02 01:45:15.276820 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:15.276785 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq" event={"ID":"d83028cc-a184-419e-b378-eb0d15e7c976","Type":"ContainerStarted","Data":"1fe6d94c2ab7c0bfe7d65acb9b35c6dbf6026e3f03818a3c5beef766386b789e"} Nov 02 01:45:15.395763 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:15.395719 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq" podStartSLOduration=15.395705095 podStartE2EDuration="15.395705095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:15.393481926 +0000 UTC m=+1232.064734616" watchObservedRunningTime="2026-11-02 01:45:15.395705095 +0000 UTC m=+1232.066957772" Nov 02 01:45:15.445140 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:15.445115 5044 scope.go:122] "RemoveContainer" containerID="4a45e14a2a418de77c2ae385b74b08407f0e4e1c2919cfe2de2c0b2e5b0220be" Nov 02 01:45:16.285592 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:16.285567 5044 generic.go:356] "Generic (PLEG): container finished" podID="d83028cc-a184-419e-b378-eb0d15e7c976" containerID="1fe6d94c2ab7c0bfe7d65acb9b35c6dbf6026e3f03818a3c5beef766386b789e" exitCode=60 Nov 02 01:45:16.286021 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:16.285652 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq" event={"ID":"d83028cc-a184-419e-b378-eb0d15e7c976","Type":"ContainerDied","Data":"1fe6d94c2ab7c0bfe7d65acb9b35c6dbf6026e3f03818a3c5beef766386b789e"} Nov 02 01:45:16.286021 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:16.285769 5044 scope.go:122] "RemoveContainer" containerID="59df9824c6dfb70aeb38011d7a06af7a7cb5c982d7127c476a6610ae6c1fdc9f" Nov 02 01:45:16.286193 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:16.286176 5044 scope.go:122] "RemoveContainer" containerID="1fe6d94c2ab7c0bfe7d65acb9b35c6dbf6026e3f03818a3c5beef766386b789e" Nov 02 01:45:16.286530 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:45:16.286506 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq_odh-ai-gateway-infra(d83028cc-a184-419e-b378-eb0d15e7c976)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq" podUID="d83028cc-a184-419e-b378-eb0d15e7c976" Nov 02 01:45:16.288665 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:16.288642 5044 generic.go:356] "Generic (PLEG): container finished" podID="d71aea2c-b510-44e6-98b8-35a4676028b1" containerID="a548044d765cc0eb4fcd69516fa0c7659bbc0b1a60e0a6a6bfed2c96ca671613" exitCode=60 Nov 02 01:45:16.288774 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:16.288732 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b" event={"ID":"d71aea2c-b510-44e6-98b8-35a4676028b1","Type":"ContainerDied","Data":"a548044d765cc0eb4fcd69516fa0c7659bbc0b1a60e0a6a6bfed2c96ca671613"} Nov 02 01:45:16.289151 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:16.289134 5044 scope.go:122] "RemoveContainer" containerID="a548044d765cc0eb4fcd69516fa0c7659bbc0b1a60e0a6a6bfed2c96ca671613" Nov 02 01:45:16.289506 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:45:16.289486 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b_odh-ai-gateway-infra(d71aea2c-b510-44e6-98b8-35a4676028b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b" podUID="d71aea2c-b510-44e6-98b8-35a4676028b1" Nov 02 01:45:16.332287 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:16.332265 5044 scope.go:122] "RemoveContainer" containerID="4a45e14a2a418de77c2ae385b74b08407f0e4e1c2919cfe2de2c0b2e5b0220be" Nov 02 01:45:16.407263 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:16.407230 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq"] Nov 02 01:45:16.445378 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:16.445351 5044 scope.go:122] "RemoveContainer" containerID="a97bb1cc9e9780d1bc714b24a9b4192c90fea678fad06711fb041d2225849e2d" Nov 02 01:45:17.300846 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:17.300808 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5" event={"ID":"ea76657a-20d2-4c01-923a-422f382b53d8","Type":"ContainerStarted","Data":"c60f6a53cded93d7d38c51b3e4ac51aacbd39c102845133478cb4c55ebe815bf"} Nov 02 01:45:17.465558 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:17.465535 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq" Nov 02 01:45:17.530708 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:17.530477 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d83028cc-a184-419e-b378-eb0d15e7c976-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d83028cc-a184-419e-b378-eb0d15e7c976-maas-api-service-ca\") pod \"d83028cc-a184-419e-b378-eb0d15e7c976\" (UID: \"d83028cc-a184-419e-b378-eb0d15e7c976\") " Nov 02 01:45:17.530708 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:17.530549 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d83028cc-a184-419e-b378-eb0d15e7c976-kube-api-access-cq727\" (UniqueName: \"kubernetes.io/projected/d83028cc-a184-419e-b378-eb0d15e7c976-kube-api-access-cq727\") pod \"d83028cc-a184-419e-b378-eb0d15e7c976\" (UID: \"d83028cc-a184-419e-b378-eb0d15e7c976\") " Nov 02 01:45:17.531354 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:17.531235 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d83028cc-a184-419e-b378-eb0d15e7c976-maas-api-service-ca" pod "d83028cc-a184-419e-b378-eb0d15e7c976" (UID: "d83028cc-a184-419e-b378-eb0d15e7c976"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:17.533199 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:17.533178 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d83028cc-a184-419e-b378-eb0d15e7c976-kube-api-access-cq727" pod "d83028cc-a184-419e-b378-eb0d15e7c976" (UID: "d83028cc-a184-419e-b378-eb0d15e7c976"). InnerVolumeSpecName "kube-api-access-cq727". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:17.533537 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:17.533499 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5" podStartSLOduration=17.533486866 podStartE2EDuration="17.533486866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:17.531091158 +0000 UTC m=+1234.202343856" watchObservedRunningTime="2026-11-02 01:45:17.533486866 +0000 UTC m=+1234.204739546" Nov 02 01:45:17.544729 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:17.544708 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b"] Nov 02 01:45:17.632851 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:17.632819 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d83028cc-a184-419e-b378-eb0d15e7c976-maas-api-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:45:17.633035 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:17.632854 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq727\" (UniqueName: \"kubernetes.io/projected/d83028cc-a184-419e-b378-eb0d15e7c976-kube-api-access-cq727\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:45:17.699646 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:17.699623 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b" Nov 02 01:45:17.733187 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:17.733157 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d71aea2c-b510-44e6-98b8-35a4676028b1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71aea2c-b510-44e6-98b8-35a4676028b1-maas-api-service-ca\") pod \"d71aea2c-b510-44e6-98b8-35a4676028b1\" (UID: \"d71aea2c-b510-44e6-98b8-35a4676028b1\") " Nov 02 01:45:17.733312 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:17.733245 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71aea2c-b510-44e6-98b8-35a4676028b1-kube-api-access-k2ll9\" (UniqueName: \"kubernetes.io/projected/d71aea2c-b510-44e6-98b8-35a4676028b1-kube-api-access-k2ll9\") pod \"d71aea2c-b510-44e6-98b8-35a4676028b1\" (UID: \"d71aea2c-b510-44e6-98b8-35a4676028b1\") " Nov 02 01:45:17.733505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:17.733481 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71aea2c-b510-44e6-98b8-35a4676028b1-maas-api-service-ca" pod "d71aea2c-b510-44e6-98b8-35a4676028b1" (UID: "d71aea2c-b510-44e6-98b8-35a4676028b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:17.733790 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:17.733763 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71aea2c-b510-44e6-98b8-35a4676028b1-maas-api-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:45:17.735228 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:17.735206 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71aea2c-b510-44e6-98b8-35a4676028b1-kube-api-access-k2ll9" pod "d71aea2c-b510-44e6-98b8-35a4676028b1" (UID: "d71aea2c-b510-44e6-98b8-35a4676028b1"). InnerVolumeSpecName "kube-api-access-k2ll9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:17.834595 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:17.834568 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2ll9\" (UniqueName: \"kubernetes.io/projected/d71aea2c-b510-44e6-98b8-35a4676028b1-kube-api-access-k2ll9\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:45:18.308283 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:18.308250 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq" Nov 02 01:45:18.308798 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:18.308255 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq" event={"ID":"d83028cc-a184-419e-b378-eb0d15e7c976","Type":"ContainerDied","Data":"f00e15b2cb3222823a482c4dde2f3ffc0711ce834bea4a89b0289c197fedb1d0"} Nov 02 01:45:18.308798 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:18.308418 5044 scope.go:122] "RemoveContainer" containerID="1fe6d94c2ab7c0bfe7d65acb9b35c6dbf6026e3f03818a3c5beef766386b789e" Nov 02 01:45:18.309736 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:18.309699 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b" event={"ID":"d71aea2c-b510-44e6-98b8-35a4676028b1","Type":"ContainerDied","Data":"757dfd45b843a8f286320873e21392e11c24793a3d77cf822f51706b734cf553"} Nov 02 01:45:18.309736 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:18.309727 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b" Nov 02 01:45:18.312571 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:18.312545 5044 generic.go:356] "Generic (PLEG): container finished" podID="ea76657a-20d2-4c01-923a-422f382b53d8" containerID="c60f6a53cded93d7d38c51b3e4ac51aacbd39c102845133478cb4c55ebe815bf" exitCode=60 Nov 02 01:45:18.312773 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:18.312632 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5" event={"ID":"ea76657a-20d2-4c01-923a-422f382b53d8","Type":"ContainerDied","Data":"c60f6a53cded93d7d38c51b3e4ac51aacbd39c102845133478cb4c55ebe815bf"} Nov 02 01:45:18.313026 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:18.313005 5044 scope.go:122] "RemoveContainer" containerID="c60f6a53cded93d7d38c51b3e4ac51aacbd39c102845133478cb4c55ebe815bf" Nov 02 01:45:18.313300 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:45:18.313276 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5_odh-ai-gateway-infra(ea76657a-20d2-4c01-923a-422f382b53d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5" podUID="ea76657a-20d2-4c01-923a-422f382b53d8" Nov 02 01:45:18.320376 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:18.320356 5044 scope.go:122] "RemoveContainer" containerID="a548044d765cc0eb4fcd69516fa0c7659bbc0b1a60e0a6a6bfed2c96ca671613" Nov 02 01:45:18.329995 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:18.329978 5044 scope.go:122] "RemoveContainer" containerID="a97bb1cc9e9780d1bc714b24a9b4192c90fea678fad06711fb041d2225849e2d" Nov 02 01:45:18.542661 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:18.542630 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5"] Nov 02 01:45:18.591781 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:18.591705 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq"] Nov 02 01:45:18.596088 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:18.596060 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-187478-29893065-c7mpq"] Nov 02 01:45:18.707270 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:18.707231 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b"] Nov 02 01:45:18.713881 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:18.713853 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f276c3-29893065-fws5b"] Nov 02 01:45:19.027402 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:19.027328 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:45:19.453341 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:19.453310 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d71aea2c-b510-44e6-98b8-35a4676028b1" path="/var/lib/kubelet/pods/d71aea2c-b510-44e6-98b8-35a4676028b1/volumes" Nov 02 01:45:19.453832 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:19.453717 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d83028cc-a184-419e-b378-eb0d15e7c976" path="/var/lib/kubelet/pods/d83028cc-a184-419e-b378-eb0d15e7c976/volumes" Nov 02 01:45:19.509062 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:19.509037 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5" Nov 02 01:45:19.553595 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:19.553566 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea76657a-20d2-4c01-923a-422f382b53d8-kube-api-access-4svt9\" (UniqueName: \"kubernetes.io/projected/ea76657a-20d2-4c01-923a-422f382b53d8-kube-api-access-4svt9\") pod \"ea76657a-20d2-4c01-923a-422f382b53d8\" (UID: \"ea76657a-20d2-4c01-923a-422f382b53d8\") " Nov 02 01:45:19.553785 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:19.553725 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea76657a-20d2-4c01-923a-422f382b53d8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea76657a-20d2-4c01-923a-422f382b53d8-maas-api-service-ca\") pod \"ea76657a-20d2-4c01-923a-422f382b53d8\" (UID: \"ea76657a-20d2-4c01-923a-422f382b53d8\") " Nov 02 01:45:19.554707 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:19.554030 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea76657a-20d2-4c01-923a-422f382b53d8-maas-api-service-ca" pod "ea76657a-20d2-4c01-923a-422f382b53d8" (UID: "ea76657a-20d2-4c01-923a-422f382b53d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:45:19.554707 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:19.554686 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea76657a-20d2-4c01-923a-422f382b53d8-maas-api-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:45:19.555907 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:19.555879 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea76657a-20d2-4c01-923a-422f382b53d8-kube-api-access-4svt9" pod "ea76657a-20d2-4c01-923a-422f382b53d8" (UID: "ea76657a-20d2-4c01-923a-422f382b53d8"). InnerVolumeSpecName "kube-api-access-4svt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:45:19.655772 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:19.655696 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4svt9\" (UniqueName: \"kubernetes.io/projected/ea76657a-20d2-4c01-923a-422f382b53d8-kube-api-access-4svt9\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:45:20.330457 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:20.330425 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5" Nov 02 01:45:20.330708 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:20.330422 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5" event={"ID":"ea76657a-20d2-4c01-923a-422f382b53d8","Type":"ContainerDied","Data":"f53409a8862de3f052b7468cd6353322897e5cf68ad794f071408bea2b1fa2e7"} Nov 02 01:45:20.330708 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:20.330576 5044 scope.go:122] "RemoveContainer" containerID="c60f6a53cded93d7d38c51b3e4ac51aacbd39c102845133478cb4c55ebe815bf" Nov 02 01:45:20.501851 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:20.501814 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5"] Nov 02 01:45:20.506464 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:20.506436 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b71ba5-29893065-4mql5"] Nov 02 01:45:21.452768 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:21.452732 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea76657a-20d2-4c01-923a-422f382b53d8" path="/var/lib/kubelet/pods/ea76657a-20d2-4c01-923a-422f382b53d8/volumes" Nov 02 01:45:23.765870 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:23.765832 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:45:24.174460 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:45:24.174363 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:45:25.589271 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:25.589205 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:45:28.591796 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:28.591742 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:45:33.370256 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:33.370219 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:45:34.429478 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.429443 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84"] Nov 02 01:45:34.430030 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430014 5044 cpu_manager.go:414] "containerMap: removing container" podUID="d71aea2c-b510-44e6-98b8-35a4676028b1" containerName="cleanup" Nov 02 01:45:34.430074 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430033 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71aea2c-b510-44e6-98b8-35a4676028b1" containerName="cleanup" Nov 02 01:45:34.430074 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430059 5044 cpu_manager.go:414] "containerMap: removing container" podUID="d83028cc-a184-419e-b378-eb0d15e7c976" containerName="cleanup" Nov 02 01:45:34.430074 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430067 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d83028cc-a184-419e-b378-eb0d15e7c976" containerName="cleanup" Nov 02 01:45:34.430175 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430075 5044 cpu_manager.go:414] "containerMap: removing container" podUID="d71aea2c-b510-44e6-98b8-35a4676028b1" containerName="cleanup" Nov 02 01:45:34.430175 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430081 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71aea2c-b510-44e6-98b8-35a4676028b1" containerName="cleanup" Nov 02 01:45:34.430175 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430088 5044 cpu_manager.go:414] "containerMap: removing container" podUID="d83028cc-a184-419e-b378-eb0d15e7c976" containerName="cleanup" Nov 02 01:45:34.430175 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430092 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d83028cc-a184-419e-b378-eb0d15e7c976" containerName="cleanup" Nov 02 01:45:34.430175 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430098 5044 cpu_manager.go:414] "containerMap: removing container" podUID="ea76657a-20d2-4c01-923a-422f382b53d8" containerName="cleanup" Nov 02 01:45:34.430175 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430102 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea76657a-20d2-4c01-923a-422f382b53d8" containerName="cleanup" Nov 02 01:45:34.430175 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430110 5044 cpu_manager.go:414] "containerMap: removing container" podUID="d83028cc-a184-419e-b378-eb0d15e7c976" containerName="cleanup" Nov 02 01:45:34.430175 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430114 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d83028cc-a184-419e-b378-eb0d15e7c976" containerName="cleanup" Nov 02 01:45:34.430175 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430132 5044 cpu_manager.go:414] "containerMap: removing container" podUID="ea76657a-20d2-4c01-923a-422f382b53d8" containerName="cleanup" Nov 02 01:45:34.430175 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430136 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea76657a-20d2-4c01-923a-422f382b53d8" containerName="cleanup" Nov 02 01:45:34.430461 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430232 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="d83028cc-a184-419e-b378-eb0d15e7c976" containerName="cleanup" Nov 02 01:45:34.430461 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430241 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71aea2c-b510-44e6-98b8-35a4676028b1" containerName="cleanup" Nov 02 01:45:34.430461 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430247 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="d83028cc-a184-419e-b378-eb0d15e7c976" containerName="cleanup" Nov 02 01:45:34.430461 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430254 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea76657a-20d2-4c01-923a-422f382b53d8" containerName="cleanup" Nov 02 01:45:34.430461 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430260 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71aea2c-b510-44e6-98b8-35a4676028b1" containerName="cleanup" Nov 02 01:45:34.430461 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430266 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="d83028cc-a184-419e-b378-eb0d15e7c976" containerName="cleanup" Nov 02 01:45:34.430461 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430283 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea76657a-20d2-4c01-923a-422f382b53d8" containerName="cleanup" Nov 02 01:45:34.430461 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.430289 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea76657a-20d2-4c01-923a-422f382b53d8" containerName="cleanup" Nov 02 01:45:34.433532 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.433514 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:34.490098 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.490067 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5888dc\"/\"kube-root-ca.crt\"" Nov 02 01:45:34.523112 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.523071 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:45:34.540772 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.540659 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5888dc\"/\"openshift-service-ca.crt\"" Nov 02 01:45:34.540966 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.540872 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5888dc\"/\"default-dockercfg-mxnhx\"" Nov 02 01:45:34.540966 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.540880 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5888dc\"/\"distinct-2-w6-5888dc-kserve-self-signed-certs\"" Nov 02 01:45:34.544216 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.544193 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84"] Nov 02 01:45:34.556314 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.556289 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-dshm\") pod \"distinct-2-w6-5888dc-kserve-844bf46b48-7km84\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:34.556416 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.556324 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-model-cache\") pod \"distinct-2-w6-5888dc-kserve-844bf46b48-7km84\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:34.556416 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.556348 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-tmp-dir\") pod \"distinct-2-w6-5888dc-kserve-844bf46b48-7km84\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:34.556549 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.556531 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ae3a465-c735-401b-a679-0a6141ca6bb9-tls-certs\") pod \"distinct-2-w6-5888dc-kserve-844bf46b48-7km84\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:34.556721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.556702 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-home\") pod \"distinct-2-w6-5888dc-kserve-844bf46b48-7km84\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:34.556837 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.556818 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lknq5\" (UniqueName: \"kubernetes.io/projected/3ae3a465-c735-401b-a679-0a6141ca6bb9-kube-api-access-lknq5\") pod \"distinct-2-w6-5888dc-kserve-844bf46b48-7km84\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:34.658004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.657969 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lknq5\" (UniqueName: \"kubernetes.io/projected/3ae3a465-c735-401b-a679-0a6141ca6bb9-kube-api-access-lknq5\") pod \"distinct-2-w6-5888dc-kserve-844bf46b48-7km84\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:34.658177 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.658031 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-dshm\") pod \"distinct-2-w6-5888dc-kserve-844bf46b48-7km84\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:34.658177 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.658054 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-model-cache\") pod \"distinct-2-w6-5888dc-kserve-844bf46b48-7km84\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:34.658177 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.658072 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-tmp-dir\") pod \"distinct-2-w6-5888dc-kserve-844bf46b48-7km84\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:34.658177 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.658152 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ae3a465-c735-401b-a679-0a6141ca6bb9-tls-certs\") pod \"distinct-2-w6-5888dc-kserve-844bf46b48-7km84\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:34.658334 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.658181 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-home\") pod \"distinct-2-w6-5888dc-kserve-844bf46b48-7km84\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:34.658556 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.658530 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-model-cache\") pod \"distinct-2-w6-5888dc-kserve-844bf46b48-7km84\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:34.658612 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.658598 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-home\") pod \"distinct-2-w6-5888dc-kserve-844bf46b48-7km84\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:34.658666 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.658629 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-tmp-dir\") pod \"distinct-2-w6-5888dc-kserve-844bf46b48-7km84\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:34.660409 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.660388 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-dshm\") pod \"distinct-2-w6-5888dc-kserve-844bf46b48-7km84\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:34.660700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.660661 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ae3a465-c735-401b-a679-0a6141ca6bb9-tls-certs\") pod \"distinct-2-w6-5888dc-kserve-844bf46b48-7km84\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:34.719832 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.719769 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lknq5\" (UniqueName: \"kubernetes.io/projected/3ae3a465-c735-401b-a679-0a6141ca6bb9-kube-api-access-lknq5\") pod \"distinct-2-w6-5888dc-kserve-844bf46b48-7km84\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:34.746316 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:34.746287 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:35.129466 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:45:35.129430 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ae3a465_c735_401b_a679_0a6141ca6bb9.slice/crio-e7f45f7d7695365da3da18a8f0c8c1ae819a38a8bb22c6a5c8c4909242a1486a WatchSource:0}: Error finding container e7f45f7d7695365da3da18a8f0c8c1ae819a38a8bb22c6a5c8c4909242a1486a: Status 404 returned error can't find the container with id e7f45f7d7695365da3da18a8f0c8c1ae819a38a8bb22c6a5c8c4909242a1486a Nov 02 01:45:35.130312 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:35.130133 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84"] Nov 02 01:45:35.436178 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:35.436085 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" event={"ID":"3ae3a465-c735-401b-a679-0a6141ca6bb9","Type":"ContainerStarted","Data":"15e823be3d44cd8f61bc792ddbe6dda0d7e51157a7611d562f610b2c72ce6402"} Nov 02 01:45:35.436178 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:35.436136 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" event={"ID":"3ae3a465-c735-401b-a679-0a6141ca6bb9","Type":"ContainerStarted","Data":"e7f45f7d7695365da3da18a8f0c8c1ae819a38a8bb22c6a5c8c4909242a1486a"} Nov 02 01:45:35.558019 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:35.557954 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podStartSLOduration=1.5579347129999999 podStartE2EDuration="1.557934713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:45:35.555204676 +0000 UTC m=+1252.226457346" watchObservedRunningTime="2026-11-02 01:45:35.557934713 +0000 UTC m=+1252.229187398" Nov 02 01:45:35.746791 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:35.746714 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:38.218778 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:38.218258 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:45:39.976807 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:39.976770 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:45:42.987467 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:42.987432 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:45:44.746872 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:44.746839 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:44.814875 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:44.814843 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:45.542848 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:45.542819 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:45:47.791558 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:47.791515 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:45:52.594072 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:52.594034 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:45:54.172816 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:45:54.172784 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:45:57.390293 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:45:57.390251 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:02.281182 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:02.281146 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:07.091703 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:07.091658 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:11.902855 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:11.902818 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:16.717635 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:16.717600 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:17.768979 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:17.768872 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:21.689645 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:21.689597 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:24.175227 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:46:24.175145 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:46:26.446505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:26.446471 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:31.283144 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.283097 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:31.518179 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.518142 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx"] Nov 02 01:46:31.518780 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.518760 5044 cpu_manager.go:414] "containerMap: removing container" podUID="ea76657a-20d2-4c01-923a-422f382b53d8" containerName="cleanup" Nov 02 01:46:31.518780 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.518779 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea76657a-20d2-4c01-923a-422f382b53d8" containerName="cleanup" Nov 02 01:46:31.518960 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.518830 5044 cpu_manager.go:414] "containerMap: removing container" podUID="d71aea2c-b510-44e6-98b8-35a4676028b1" containerName="cleanup" Nov 02 01:46:31.518960 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.518838 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71aea2c-b510-44e6-98b8-35a4676028b1" containerName="cleanup" Nov 02 01:46:31.518960 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.518958 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71aea2c-b510-44e6-98b8-35a4676028b1" containerName="cleanup" Nov 02 01:46:31.522456 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.522434 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:31.589569 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.589472 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-dshm\") pod \"distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:31.589569 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.589542 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-home\") pod \"distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:31.589834 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.589730 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-tls-certs\") pod \"distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:31.589834 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.589774 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-tmp-dir\") pod \"distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:31.589959 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.589915 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bwll\" (UniqueName: \"kubernetes.io/projected/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-kube-api-access-5bwll\") pod \"distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:31.590048 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.590031 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-model-cache\") pod \"distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:31.603461 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.603432 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:31.639004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.638977 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-88e76d\"/\"openshift-service-ca.crt\"" Nov 02 01:46:31.639004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.638981 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-88e76d\"/\"default-dockercfg-2rrmt\"" Nov 02 01:46:31.639004 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.638987 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-88e76d\"/\"kube-root-ca.crt\"" Nov 02 01:46:31.639273 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.639123 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-88e76d\"/\"distinct-2-w3-88e76d-kserve-self-signed-certs\"" Nov 02 01:46:31.642358 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.642335 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx"] Nov 02 01:46:31.691208 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.691162 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-model-cache\") pod \"distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:31.691436 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.691297 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-dshm\") pod \"distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:31.691436 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.691353 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-home\") pod \"distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:31.691582 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.691458 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-tls-certs\") pod \"distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:31.691582 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.691505 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-tmp-dir\") pod \"distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:31.691582 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.691553 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bwll\" (UniqueName: \"kubernetes.io/projected/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-kube-api-access-5bwll\") pod \"distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:31.692379 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.692351 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-model-cache\") pod \"distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:31.693441 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.693393 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-tmp-dir\") pod \"distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:31.694706 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.694682 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-home\") pod \"distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:31.694955 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.694931 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-dshm\") pod \"distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:31.695769 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.695749 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-tls-certs\") pod \"distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:31.754297 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.754257 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bwll\" (UniqueName: \"kubernetes.io/projected/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-kube-api-access-5bwll\") pod \"distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:31.835155 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:31.835121 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:32.226159 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:32.226133 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx"] Nov 02 01:46:32.227081 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:46:32.227056 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfd6f3e6_f1f2_4f85_85a4_a2378ddd2cbc.slice/crio-7955aac7fb88a6a1f006209dcd51b3f0c1324c525e0ae3d4eb8f670b59995514 WatchSource:0}: Error finding container 7955aac7fb88a6a1f006209dcd51b3f0c1324c525e0ae3d4eb8f670b59995514: Status 404 returned error can't find the container with id 7955aac7fb88a6a1f006209dcd51b3f0c1324c525e0ae3d4eb8f670b59995514 Nov 02 01:46:32.843767 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:32.843729 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" event={"ID":"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc","Type":"ContainerStarted","Data":"3727e355a31ffcf685f5885504d693553743c45d6e5372ea9d7b46e2031ab02e"} Nov 02 01:46:32.843767 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:32.843775 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" event={"ID":"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc","Type":"ContainerStarted","Data":"7955aac7fb88a6a1f006209dcd51b3f0c1324c525e0ae3d4eb8f670b59995514"} Nov 02 01:46:32.965147 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:32.965091 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podStartSLOduration=1.965077593 podStartE2EDuration="1.965077593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:46:32.962499975 +0000 UTC m=+1309.633752656" watchObservedRunningTime="2026-11-02 01:46:32.965077593 +0000 UTC m=+1309.636330288" Nov 02 01:46:33.835874 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:33.835841 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:36.031710 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:36.031663 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:37.796564 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:37.796534 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:40.319122 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:40.319081 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:40.861193 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:40.861151 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:41.835812 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:41.835753 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:41.848536 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:41.848514 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:41.924088 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:41.924064 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:46:43.941393 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:43.941360 5044 generic.go:356] "Generic (PLEG): container finished" podID="f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1" containerID="eec6f1c6acd6ff69cb906b6fd10aed67859349f8de49ab55e10d800f277e4b9c" exitCode=0 Nov 02 01:46:43.941801 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:43.941434 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" event={"ID":"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1","Type":"ContainerDied","Data":"eec6f1c6acd6ff69cb906b6fd10aed67859349f8de49ab55e10d800f277e4b9c"} Nov 02 01:46:44.921526 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:44.921493 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6"] Nov 02 01:46:44.925785 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:44.925761 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:44.969543 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:44.969013 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-tmp-dir\") pod \"distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:44.969543 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:44.969108 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39920d19-93e7-46ad-a3a4-90a51b766ab3-tls-certs\") pod \"distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:44.969543 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:44.969225 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-model-cache\") pod \"distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:44.969543 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:44.969290 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-dshm\") pod \"distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:44.969543 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:44.969314 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7vv9\" (UniqueName: \"kubernetes.io/projected/39920d19-93e7-46ad-a3a4-90a51b766ab3-kube-api-access-n7vv9\") pod \"distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:44.969543 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:44.969360 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-home\") pod \"distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:45.007146 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.006902 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:45.034294 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.031843 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-36a1e0\"/\"default-dockercfg-t2p55\"" Nov 02 01:46:45.034294 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.031965 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-36a1e0\"/\"distinct-2-w2-36a1e0-kserve-self-signed-certs\"" Nov 02 01:46:45.034294 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.032634 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-36a1e0\"/\"openshift-service-ca.crt\"" Nov 02 01:46:45.034294 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.033534 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-36a1e0\"/\"kube-root-ca.crt\"" Nov 02 01:46:45.037638 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.037613 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6"] Nov 02 01:46:45.078713 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.074525 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-model-cache\") pod \"distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:45.078713 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.074611 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-dshm\") pod \"distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:45.078713 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.074638 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7vv9\" (UniqueName: \"kubernetes.io/projected/39920d19-93e7-46ad-a3a4-90a51b766ab3-kube-api-access-n7vv9\") pod \"distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:45.078713 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.074723 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-home\") pod \"distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:45.078713 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.074795 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-tmp-dir\") pod \"distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:45.078713 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.074867 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39920d19-93e7-46ad-a3a4-90a51b766ab3-tls-certs\") pod \"distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:45.078713 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.076000 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-model-cache\") pod \"distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:45.078713 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.076531 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-tmp-dir\") pod \"distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:45.078713 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.076806 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-home\") pod \"distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:45.087395 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.087365 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39920d19-93e7-46ad-a3a4-90a51b766ab3-tls-certs\") pod \"distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:45.098741 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.091623 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-dshm\") pod \"distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:45.138171 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.138145 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7vv9\" (UniqueName: \"kubernetes.io/projected/39920d19-93e7-46ad-a3a4-90a51b766ab3-kube-api-access-n7vv9\") pod \"distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:45.161964 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.161935 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" Nov 02 01:46:45.238952 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.238874 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:45.280211 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.280171 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-archives-path\") pod \"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1\" (UID: \"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1\") " Nov 02 01:46:45.280393 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.280292 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-service-ca-bundle\") pod \"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1\" (UID: \"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1\") " Nov 02 01:46:45.280393 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.280386 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-trusted-ca-bundle\") pod \"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1\" (UID: \"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1\") " Nov 02 01:46:45.280835 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.280472 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-kube-api-access-wh889\" (UniqueName: \"kubernetes.io/projected/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-kube-api-access-wh889\") pod \"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1\" (UID: \"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1\") " Nov 02 01:46:45.290469 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.290434 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-service-ca-bundle" pod "f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1" (UID: "f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:45.291596 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.291554 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-trusted-ca-bundle" pod "f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1" (UID: "f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:46:45.300768 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.293299 5044 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-trusted-ca-bundle\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:46:45.300768 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.293330 5044 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-service-ca-bundle\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:46:45.321348 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.321289 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-archives-path" pod "f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1" (UID: "f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:46:45.347202 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.345005 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-kube-api-access-wh889" pod "f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1" (UID: "f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1"). InnerVolumeSpecName "kube-api-access-wh889". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:46:45.395139 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.395089 5044 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-archives-path\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:46:45.395139 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.395119 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh889\" (UniqueName: \"kubernetes.io/projected/f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1-kube-api-access-wh889\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:46:45.672959 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.672933 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6"] Nov 02 01:46:45.675432 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:46:45.675401 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39920d19_93e7_46ad_a3a4_90a51b766ab3.slice/crio-7f5f700fae5dc182578515bc8b94f9bdefe5b28c5834e01c75de9a60a4f9c4b8 WatchSource:0}: Error finding container 7f5f700fae5dc182578515bc8b94f9bdefe5b28c5834e01c75de9a60a4f9c4b8: Status 404 returned error can't find the container with id 7f5f700fae5dc182578515bc8b94f9bdefe5b28c5834e01c75de9a60a4f9c4b8 Nov 02 01:46:45.881767 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.881729 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:45.958739 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.958637 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" event={"ID":"39920d19-93e7-46ad-a3a4-90a51b766ab3","Type":"ContainerStarted","Data":"45d7a0f24c32cdc4bb1cbcc70f79706c3179a62945f3762b883381797064e4fd"} Nov 02 01:46:45.958739 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.958713 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" event={"ID":"39920d19-93e7-46ad-a3a4-90a51b766ab3","Type":"ContainerStarted","Data":"7f5f700fae5dc182578515bc8b94f9bdefe5b28c5834e01c75de9a60a4f9c4b8"} Nov 02 01:46:45.960101 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.960076 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" Nov 02 01:46:45.960219 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.960076 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-qwlzg-9pp4j" event={"ID":"f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1","Type":"ContainerDied","Data":"e48040dc3a48dc8ca90588425d4478197fb655723cd4239f9eecf2c3b21f3c28"} Nov 02 01:46:45.960262 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:45.960223 5044 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e48040dc3a48dc8ca90588425d4478197fb655723cd4239f9eecf2c3b21f3c28" Nov 02 01:46:46.132037 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:46.131988 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podStartSLOduration=2.131973409 podStartE2EDuration="2.131973409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:46:46.08174359 +0000 UTC m=+1322.752996275" watchObservedRunningTime="2026-11-02 01:46:46.131973409 +0000 UTC m=+1322.803226145" Nov 02 01:46:46.240025 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:46.239938 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:46.916957 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:46.916920 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:47.059092 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:47.059059 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:50.670935 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:50.670901 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:54.173325 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:46:54.173289 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:46:55.239394 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.239348 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:55.253150 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.253126 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:55.426790 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.426753 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg"] Nov 02 01:46:55.427377 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.427357 5044 cpu_manager.go:414] "containerMap: removing container" podUID="f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1" containerName="insights-gathering" Nov 02 01:46:55.427480 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.427378 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1" containerName="insights-gathering" Nov 02 01:46:55.427580 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.427566 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0cdd1f8-c3bb-4387-8a14-c7b62b0641f1" containerName="insights-gathering" Nov 02 01:46:55.431134 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.431114 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.466352 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.466323 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:55.482431 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.482402 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg"] Nov 02 01:46:55.534710 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.526384 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.534710 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.526450 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.534710 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.526477 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.534710 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.526504 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.534710 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.526601 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.534710 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.526640 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfbjr\" (UniqueName: \"kubernetes.io/projected/e541a05c-0d9f-420f-af63-638e2642c6aa-kube-api-access-jfbjr\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.534710 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.526659 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e541a05c-0d9f-420f-af63-638e2642c6aa-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.534710 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.526717 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.534710 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.527062 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.628077 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.628048 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.628295 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.628100 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfbjr\" (UniqueName: \"kubernetes.io/projected/e541a05c-0d9f-420f-af63-638e2642c6aa-kube-api-access-jfbjr\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.628295 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.628122 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e541a05c-0d9f-420f-af63-638e2642c6aa-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.628295 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.628158 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.628295 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.628276 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.628529 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.628309 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.628529 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.628334 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.628529 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.628358 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.628529 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.628382 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.628784 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.628719 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.628866 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.628840 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.628931 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.628867 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.628931 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.628913 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.629035 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.629007 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e541a05c-0d9f-420f-af63-638e2642c6aa-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.631043 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.631024 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.631302 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.631282 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.690068 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.690038 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.690293 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.690260 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfbjr\" (UniqueName: \"kubernetes.io/projected/e541a05c-0d9f-420f-af63-638e2642c6aa-kube-api-access-jfbjr\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:55.748356 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:55.748325 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:56.039616 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:56.039589 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:46:56.180283 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:56.180253 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg"] Nov 02 01:46:56.182411 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:46:56.182366 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode541a05c_0d9f_420f_af63_638e2642c6aa.slice/crio-5f22453872807d7e1f711a2aba3fc9a0520daa1f920f4ad118826cc406635df2 WatchSource:0}: Error finding container 5f22453872807d7e1f711a2aba3fc9a0520daa1f920f4ad118826cc406635df2: Status 404 returned error can't find the container with id 5f22453872807d7e1f711a2aba3fc9a0520daa1f920f4ad118826cc406635df2 Nov 02 01:46:56.185474 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:56.185437 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:46:56.185587 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:56.185522 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:46:56.185587 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:56.185566 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:46:57.036397 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:57.036360 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" event={"ID":"e541a05c-0d9f-420f-af63-638e2642c6aa","Type":"ContainerStarted","Data":"04d69b4c5a6c8affc1a365226c709c8731d992c16f31e32bc2de72122fdf62bb"} Nov 02 01:46:57.036841 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:57.036421 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" event={"ID":"e541a05c-0d9f-420f-af63-638e2642c6aa","Type":"ContainerStarted","Data":"5f22453872807d7e1f711a2aba3fc9a0520daa1f920f4ad118826cc406635df2"} Nov 02 01:46:57.259572 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:57.259518 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" podStartSLOduration=2.259502446 podStartE2EDuration="2.259502446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:46:57.208287456 +0000 UTC m=+1333.879540149" watchObservedRunningTime="2026-11-02 01:46:57.259502446 +0000 UTC m=+1333.930755118" Nov 02 01:46:57.748895 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:57.748851 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:46:57.750501 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:57.750474 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 01:46:57.750651 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:57.750535 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" podUID="e541a05c-0d9f-420f-af63-638e2642c6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 01:46:58.203199 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:58.203167 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:46:58.748639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:58.748603 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 01:46:58.748872 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:58.748656 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" podUID="e541a05c-0d9f-420f-af63-638e2642c6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 01:46:59.749699 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:59.749644 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 01:46:59.750083 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:46:59.749713 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" podUID="e541a05c-0d9f-420f-af63-638e2642c6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 01:47:00.387002 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:00.386964 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:47:00.749246 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:00.749155 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 01:47:00.749246 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:00.749219 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" podUID="e541a05c-0d9f-420f-af63-638e2642c6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 01:47:01.749037 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:01.749000 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 01:47:01.749457 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:01.749052 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" podUID="e541a05c-0d9f-420f-af63-638e2642c6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 01:47:02.749436 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:02.749398 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 01:47:02.749856 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:02.749463 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" podUID="e541a05c-0d9f-420f-af63-638e2642c6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 01:47:03.754138 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:03.754111 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:47:03.754600 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:03.754587 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:47:03.755419 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:03.755399 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:47:05.176304 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:05.176274 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:47:09.924733 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:09.924694 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:47:10.228750 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.227788 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr"] Nov 02 01:47:10.234507 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.234477 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.338578 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.338544 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr"] Nov 02 01:47:10.350000 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.349970 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.350144 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.350020 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.350202 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.350163 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33422040-a460-438d-a99b-65df8511eeb0-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.350251 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.350234 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.350318 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.350301 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.350356 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.350342 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33422040-a460-438d-a99b-65df8511eeb0-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.350415 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.350403 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33422040-a460-438d-a99b-65df8511eeb0-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.350451 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.350441 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.350499 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.350488 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grgwl\" (UniqueName: \"kubernetes.io/projected/33422040-a460-438d-a99b-65df8511eeb0-kube-api-access-grgwl\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.452229 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.452194 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.452407 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.452290 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33422040-a460-438d-a99b-65df8511eeb0-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.452407 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.452343 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.452407 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.452374 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.452648 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.452621 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-istio-data\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.452740 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.452690 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33422040-a460-438d-a99b-65df8511eeb0-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.452740 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.452728 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-workload-certs\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.452806 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.452742 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33422040-a460-438d-a99b-65df8511eeb0-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.452806 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.452795 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.452889 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.452819 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grgwl\" (UniqueName: \"kubernetes.io/projected/33422040-a460-438d-a99b-65df8511eeb0-kube-api-access-grgwl\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.452946 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.452898 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.452998 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.452981 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-workload-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.453329 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.453306 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-credential-socket\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.453455 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.453351 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33422040-a460-438d-a99b-65df8511eeb0-istiod-ca-cert\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.455396 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.455366 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-istio-envoy\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.455505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.455432 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33422040-a460-438d-a99b-65df8511eeb0-istio-podinfo\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.514745 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.514714 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33422040-a460-438d-a99b-65df8511eeb0-istio-token\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.514745 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.514734 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grgwl\" (UniqueName: \"kubernetes.io/projected/33422040-a460-438d-a99b-65df8511eeb0-kube-api-access-grgwl\") pod \"e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.553451 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.553427 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:10.988430 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.988397 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr"] Nov 02 01:47:10.989015 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:47:10.988993 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33422040_a460_438d_a99b_65df8511eeb0.slice/crio-6af1911e87de79cba884136297b8bbf8af62136b81431af96c978a85d5471fdd WatchSource:0}: Error finding container 6af1911e87de79cba884136297b8bbf8af62136b81431af96c978a85d5471fdd: Status 404 returned error can't find the container with id 6af1911e87de79cba884136297b8bbf8af62136b81431af96c978a85d5471fdd Nov 02 01:47:10.991907 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.991877 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:47:10.991994 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.991942 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:47:10.991994 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:10.991972 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:47:11.136975 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:11.136938 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" event={"ID":"33422040-a460-438d-a99b-65df8511eeb0","Type":"ContainerStarted","Data":"5e577852e5c654d4d60d03cf1ff74476c5bfaebb3f59ec37eceede6e403e239d"} Nov 02 01:47:11.137149 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:11.136996 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" event={"ID":"33422040-a460-438d-a99b-65df8511eeb0","Type":"ContainerStarted","Data":"6af1911e87de79cba884136297b8bbf8af62136b81431af96c978a85d5471fdd"} Nov 02 01:47:11.272098 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:11.272035 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" podStartSLOduration=1.272015295 podStartE2EDuration="1.272015295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:11.268124698 +0000 UTC m=+1347.939377383" watchObservedRunningTime="2026-11-02 01:47:11.272015295 +0000 UTC m=+1347.943267978" Nov 02 01:47:11.554040 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:11.553962 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:12.555906 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:12.555875 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 01:47:12.556300 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:12.555923 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" podUID="33422040-a460-438d-a99b-65df8511eeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 01:47:13.554371 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:13.554338 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 01:47:13.554548 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:13.554387 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" podUID="33422040-a460-438d-a99b-65df8511eeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 01:47:14.555327 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:14.555291 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 01:47:14.555923 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:14.555892 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" podUID="33422040-a460-438d-a99b-65df8511eeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 01:47:15.554478 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:15.554008 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 01:47:15.554478 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:15.554073 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" podUID="33422040-a460-438d-a99b-65df8511eeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 01:47:15.786907 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:15.786860 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:15.844382 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:15.844250 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:15.884814 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:15.884766 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:15.950809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:15.950760 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:15.969721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:15.969662 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:16.002812 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.002765 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:16.038965 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.038923 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:16.110360 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.110255 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:47:16.204428 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.204384 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:16.239817 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.239762 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:16.318886 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.318823 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:16.390749 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.384969 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:16.440020 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.439976 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:16.459706 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.444829 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 02 01:47:16.532694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.516145 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:16.548718 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.546160 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:16.548718 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.546550 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:16.657755 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.591829 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:16.657755 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.592176 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 02 01:47:16.657755 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.623968 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 01:47:16.657755 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.624027 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" podUID="33422040-a460-438d-a99b-65df8511eeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 01:47:16.789790 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.789750 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 01:47:16.790663 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.790643 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:47:16.890694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.890009 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:16.890694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.890482 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:47:16.951958 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.951860 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:16.952580 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.952557 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:47:16.976463 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.974790 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:16.976463 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:16.975194 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:47:17.018222 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:17.018184 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 01:47:17.024730 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:17.021146 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:47:17.163745 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:17.045012 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:17.168937 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:17.168903 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:47:17.224513 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:17.224386 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:47:17.280149 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:17.280122 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:47:17.309732 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:17.308948 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:47:17.309732 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:17.309017 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:47:17.323927 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:17.323239 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 02 01:47:17.324518 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:17.324174 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:47:17.326970 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:17.326651 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:47:17.328627 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:17.328610 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:47:17.349734 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:17.349711 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:47:17.555141 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:17.555115 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 01:47:17.555303 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:17.555161 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" podUID="33422040-a460-438d-a99b-65df8511eeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 01:47:18.553808 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:18.553775 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 01:47:18.554198 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:18.553824 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" podUID="33422040-a460-438d-a99b-65df8511eeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 01:47:19.559216 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:19.559184 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:19.559639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:19.559554 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:19.560268 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:19.560252 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:47:21.390695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:21.390652 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:47:24.173594 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:47:24.173554 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:47:24.604148 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.604108 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4"] Nov 02 01:47:24.609252 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.609230 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.654107 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.654082 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd"] Nov 02 01:47:24.657955 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.657940 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.690965 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.690925 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.691116 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.690977 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.691116 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.691048 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.691234 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.691186 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.691295 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.691235 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29375ec5-a360-4efd-8a4d-9590d0337a2b-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.691295 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.691267 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.691398 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.691306 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.691398 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.691334 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg79k\" (UniqueName: \"kubernetes.io/projected/29375ec5-a360-4efd-8a4d-9590d0337a2b-kube-api-access-lg79k\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.691398 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.691376 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.702003 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.701977 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-187478-openshift-default-dockercfg-gsqtc\"" Nov 02 01:47:24.707443 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.707414 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4"] Nov 02 01:47:24.719268 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.719247 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd"] Nov 02 01:47:24.793001 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.792959 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.793208 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.793036 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.793208 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.793081 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.793208 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.793164 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03580e97-7ea6-4a50-a1dd-640da600885c-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.793370 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.793234 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.793424 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.793370 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9cs6\" (UniqueName: \"kubernetes.io/projected/03580e97-7ea6-4a50-a1dd-640da600885c-kube-api-access-l9cs6\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.793424 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.793389 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.793525 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.793447 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.793525 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.793473 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.793525 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.793488 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.793702 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.793570 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.793702 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.793615 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.793702 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.793642 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03580e97-7ea6-4a50-a1dd-640da600885c-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.793861 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.793720 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29375ec5-a360-4efd-8a4d-9590d0337a2b-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.793861 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.793748 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.793861 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.793778 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.793861 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.793786 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.793861 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.793806 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03580e97-7ea6-4a50-a1dd-640da600885c-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.793861 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.793842 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.794159 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.793867 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg79k\" (UniqueName: \"kubernetes.io/projected/29375ec5-a360-4efd-8a4d-9590d0337a2b-kube-api-access-lg79k\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.794159 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.793872 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.794487 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.794461 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.794639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.794471 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29375ec5-a360-4efd-8a4d-9590d0337a2b-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.796413 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.796384 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.796846 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.796827 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.862875 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.862800 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.863040 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.862894 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg79k\" (UniqueName: \"kubernetes.io/projected/29375ec5-a360-4efd-8a4d-9590d0337a2b-kube-api-access-lg79k\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.895346 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.895317 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.895498 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.895351 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03580e97-7ea6-4a50-a1dd-640da600885c-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.895498 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.895431 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.895498 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.895474 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.895727 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.895507 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03580e97-7ea6-4a50-a1dd-640da600885c-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.895727 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.895712 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.895860 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.895809 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.895918 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.895882 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.896142 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.896117 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9cs6\" (UniqueName: \"kubernetes.io/projected/03580e97-7ea6-4a50-a1dd-640da600885c-kube-api-access-l9cs6\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.896284 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.896165 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.896284 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.896233 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.896284 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.896277 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03580e97-7ea6-4a50-a1dd-640da600885c-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.896467 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.896281 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03580e97-7ea6-4a50-a1dd-640da600885c-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.896602 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.896579 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.898245 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.898217 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.898415 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.898394 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03580e97-7ea6-4a50-a1dd-640da600885c-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.925447 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.925422 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:24.958235 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.958202 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9cs6\" (UniqueName: \"kubernetes.io/projected/03580e97-7ea6-4a50-a1dd-640da600885c-kube-api-access-l9cs6\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.958589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.958568 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03580e97-7ea6-4a50-a1dd-640da600885c-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:24.973821 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:24.973740 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:25.361438 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:25.361408 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4"] Nov 02 01:47:25.362131 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:47:25.362097 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29375ec5_a360_4efd_8a4d_9590d0337a2b.slice/crio-ae8645a267e7baf27e39ad9098eb17a092282617334ae96d854efd49d430375b WatchSource:0}: Error finding container ae8645a267e7baf27e39ad9098eb17a092282617334ae96d854efd49d430375b: Status 404 returned error can't find the container with id ae8645a267e7baf27e39ad9098eb17a092282617334ae96d854efd49d430375b Nov 02 01:47:25.365069 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:25.365034 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:47:25.365185 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:25.365112 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:47:25.365185 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:25.365160 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:47:25.415888 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:25.415863 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd"] Nov 02 01:47:25.416946 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:47:25.416922 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03580e97_7ea6_4a50_a1dd_640da600885c.slice/crio-bdb890105b15fefb8a8c69ffe27cfb7622f0caf841e6b104ef107874be33d4a1 WatchSource:0}: Error finding container bdb890105b15fefb8a8c69ffe27cfb7622f0caf841e6b104ef107874be33d4a1: Status 404 returned error can't find the container with id bdb890105b15fefb8a8c69ffe27cfb7622f0caf841e6b104ef107874be33d4a1 Nov 02 01:47:25.419412 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:25.419377 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:47:25.419484 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:25.419449 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:47:25.419532 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:25.419487 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:47:26.325108 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:26.325076 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:47:26.333902 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:26.333792 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" event={"ID":"03580e97-7ea6-4a50-a1dd-640da600885c","Type":"ContainerStarted","Data":"c83336d6848b9edb5cc40289869fdf5a120713fa0e701a9e75f90bc82364f8f4"} Nov 02 01:47:26.333902 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:26.333866 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" event={"ID":"03580e97-7ea6-4a50-a1dd-640da600885c","Type":"ContainerStarted","Data":"bdb890105b15fefb8a8c69ffe27cfb7622f0caf841e6b104ef107874be33d4a1"} Nov 02 01:47:26.337298 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:26.337219 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" event={"ID":"29375ec5-a360-4efd-8a4d-9590d0337a2b","Type":"ContainerStarted","Data":"d506aba1241de5f981a53b43a231db8b5c8da42b8b323778b0c59cb3467ef5e9"} Nov 02 01:47:26.337298 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:26.337263 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" event={"ID":"29375ec5-a360-4efd-8a4d-9590d0337a2b","Type":"ContainerStarted","Data":"ae8645a267e7baf27e39ad9098eb17a092282617334ae96d854efd49d430375b"} Nov 02 01:47:26.598581 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:26.598447 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" podStartSLOduration=2.598423499 podStartE2EDuration="2.598423499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:26.513381876 +0000 UTC m=+1363.184634561" watchObservedRunningTime="2026-11-02 01:47:26.598423499 +0000 UTC m=+1363.269676180" Nov 02 01:47:26.648109 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:26.648055 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" podStartSLOduration=2.6480386769999997 podStartE2EDuration="2.648038677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:26.646043545 +0000 UTC m=+1363.317296221" watchObservedRunningTime="2026-11-02 01:47:26.648038677 +0000 UTC m=+1363.319291357" Nov 02 01:47:26.926062 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:26.925955 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:26.927832 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:26.927806 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 01:47:26.927975 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:26.927852 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" podUID="29375ec5-a360-4efd-8a4d-9590d0337a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 01:47:26.981694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:26.981071 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:26.987695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:26.984813 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 01:47:26.987695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:26.984871 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" podUID="03580e97-7ea6-4a50-a1dd-640da600885c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 01:47:27.926609 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:27.926570 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 01:47:27.927043 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:27.926630 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" podUID="29375ec5-a360-4efd-8a4d-9590d0337a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 01:47:27.982694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:27.978072 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 01:47:27.982694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:27.978135 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" podUID="03580e97-7ea6-4a50-a1dd-640da600885c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 01:47:28.926111 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:28.926072 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 01:47:28.926298 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:28.926124 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" podUID="29375ec5-a360-4efd-8a4d-9590d0337a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 01:47:28.984694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:28.984048 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 01:47:28.984694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:28.984102 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" podUID="03580e97-7ea6-4a50-a1dd-640da600885c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 01:47:29.925823 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:29.925786 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 01:47:29.926047 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:29.925844 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" podUID="29375ec5-a360-4efd-8a4d-9590d0337a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 01:47:29.979695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:29.975019 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 01:47:29.979695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:29.975079 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" podUID="03580e97-7ea6-4a50-a1dd-640da600885c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 01:47:30.926218 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:30.926185 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 01:47:30.926631 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:30.926247 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" podUID="29375ec5-a360-4efd-8a4d-9590d0337a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 01:47:30.980693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:30.980034 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 01:47:30.980693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:30.980099 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" podUID="03580e97-7ea6-4a50-a1dd-640da600885c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 01:47:31.306690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:31.304247 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:47:31.926687 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:31.926640 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 01:47:31.927072 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:31.926721 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" podUID="29375ec5-a360-4efd-8a4d-9590d0337a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 01:47:31.974986 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:31.974951 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 01:47:31.975179 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:31.975009 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" podUID="03580e97-7ea6-4a50-a1dd-640da600885c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 01:47:32.934237 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:32.934202 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:32.935263 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:32.935183 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:32.936932 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:32.936235 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 01:47:32.988008 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:32.987981 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:32.988877 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:32.988723 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:32.989547 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:32.989519 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:47:36.079450 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:36.079408 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:47:39.170485 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.170449 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z"] Nov 02 01:47:39.175060 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.175042 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.257737 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.257695 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.257918 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.257787 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4wjc\" (UniqueName: \"kubernetes.io/projected/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-kube-api-access-r4wjc\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.258033 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.258004 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.258097 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.258087 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.258164 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.258147 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.258229 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.258193 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.258297 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.258239 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.258354 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.258319 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.258410 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.258376 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.289346 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.289308 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z"] Nov 02 01:47:39.359996 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.359951 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.359996 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.360003 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.360253 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.360030 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.360253 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.360058 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.360253 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.360080 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.360253 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.360213 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.360464 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.360279 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.360464 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.360346 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4wjc\" (UniqueName: \"kubernetes.io/projected/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-kube-api-access-r4wjc\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.360566 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.360465 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.360566 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.360497 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.360566 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.360524 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.360814 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.360789 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.360870 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.360789 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.360924 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.360903 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.363165 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.363140 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.363541 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.363517 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.423977 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.423914 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.424086 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.424006 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4wjc\" (UniqueName: \"kubernetes.io/projected/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-kube-api-access-r4wjc\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.490180 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.490145 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:39.932788 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.932753 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z"] Nov 02 01:47:39.968743 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:47:39.968712 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93d45d5_23dc_49ca_a2f4_c41ebdd0c15e.slice/crio-6cfb6cbd8ca353e13600f862826292f3d9b0e3373a4149ed50fc71a8622b5e47 WatchSource:0}: Error finding container 6cfb6cbd8ca353e13600f862826292f3d9b0e3373a4149ed50fc71a8622b5e47: Status 404 returned error can't find the container with id 6cfb6cbd8ca353e13600f862826292f3d9b0e3373a4149ed50fc71a8622b5e47 Nov 02 01:47:39.971501 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.971468 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:47:39.971600 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.971540 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:47:39.971600 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:39.971568 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:47:40.195027 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.194951 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw"] Nov 02 01:47:40.200212 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.200190 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.258953 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.258920 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw"] Nov 02 01:47:40.273117 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.273080 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.273278 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.273189 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.273278 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.273265 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.273391 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.273332 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.273391 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.273374 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9td42\" (UniqueName: \"kubernetes.io/projected/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-kube-api-access-9td42\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.273508 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.273440 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.273565 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.273514 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.273565 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.273541 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.273696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.273565 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.380810 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.380767 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.380953 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.380823 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.380953 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.380866 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.380953 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.380934 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.381144 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.381036 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.381144 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.381113 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.381252 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.381181 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.381252 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.381227 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9td42\" (UniqueName: \"kubernetes.io/projected/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-kube-api-access-9td42\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.381353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.381302 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.381353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.381314 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.381462 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.381380 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.381936 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.381629 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.382232 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.382208 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.382741 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.382714 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.384510 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.384486 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.384839 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.384818 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.443930 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.443902 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.444412 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.444390 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9td42\" (UniqueName: \"kubernetes.io/projected/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-kube-api-access-9td42\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.508555 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.508496 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" event={"ID":"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e","Type":"ContainerStarted","Data":"a9a76497fb4f0664d8b59400eaa47cb250978aaec2df521d33bf7a5ba12cb763"} Nov 02 01:47:40.508723 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.508557 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" event={"ID":"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e","Type":"ContainerStarted","Data":"6cfb6cbd8ca353e13600f862826292f3d9b0e3373a4149ed50fc71a8622b5e47"} Nov 02 01:47:40.516054 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.516033 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:40.634915 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.634848 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" podStartSLOduration=1.634828289 podStartE2EDuration="1.634828289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:40.630208701 +0000 UTC m=+1377.301461383" watchObservedRunningTime="2026-11-02 01:47:40.634828289 +0000 UTC m=+1377.306080970" Nov 02 01:47:40.679124 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.679083 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:47:40.983968 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.983178 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw"] Nov 02 01:47:40.997693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.989258 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:47:40.998180 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.998149 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:47:40.998251 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.998216 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:47:40.998251 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:40.998246 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:47:41.493693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:41.493433 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:41.493693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:41.493555 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 02 01:47:41.493693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:41.493595 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" podUID="c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 02 01:47:41.519755 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:41.519707 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" event={"ID":"25c9e11e-b82c-4e33-81bb-1ccae0588c2d","Type":"ContainerStarted","Data":"84ca97d6aef5c1718e897f117e436a737109df42e4486966e9497f3dc89d1b46"} Nov 02 01:47:41.519916 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:41.519782 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" event={"ID":"25c9e11e-b82c-4e33-81bb-1ccae0588c2d","Type":"ContainerStarted","Data":"169fed294880ec6f0defd2876310bd0e2e6430e9965c165318a90c8677694b62"} Nov 02 01:47:42.495696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:42.492959 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 02 01:47:42.495696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:42.493015 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" podUID="c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 02 01:47:42.518272 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:42.518236 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:42.528040 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:42.527837 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 02 01:47:42.528040 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:42.527902 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" podUID="25c9e11e-b82c-4e33-81bb-1ccae0588c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 02 01:47:43.492694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:43.492364 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 02 01:47:43.492694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:43.492424 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" podUID="c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 02 01:47:43.521702 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:43.518513 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 02 01:47:43.521702 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:43.518570 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" podUID="25c9e11e-b82c-4e33-81bb-1ccae0588c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 02 01:47:44.491694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:44.490955 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 02 01:47:44.491694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:44.491013 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" podUID="c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 02 01:47:44.519729 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:44.518088 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 02 01:47:44.519729 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:44.518156 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" podUID="25c9e11e-b82c-4e33-81bb-1ccae0588c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 02 01:47:45.494695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:45.491528 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 02 01:47:45.494695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:45.491581 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" podUID="c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 02 01:47:45.525709 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:45.525660 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 02 01:47:45.525995 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:45.525960 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" podUID="25c9e11e-b82c-4e33-81bb-1ccae0588c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 02 01:47:46.178710 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:46.172210 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:47:46.498777 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:46.490460 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 02 01:47:46.498777 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:46.490516 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" podUID="c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 02 01:47:46.516491 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:46.516461 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 02 01:47:46.516636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:46.516513 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" podUID="25c9e11e-b82c-4e33-81bb-1ccae0588c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 02 01:47:47.516477 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:47.516428 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 02 01:47:47.517028 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:47.516496 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" podUID="25c9e11e-b82c-4e33-81bb-1ccae0588c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 02 01:47:47.774104 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:47.774009 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:47.774455 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:47.774431 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:47.775324 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:47.775299 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:47:47.956739 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:47.954250 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" podStartSLOduration=7.954229574 podStartE2EDuration="7.954229574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:41.690978834 +0000 UTC m=+1378.362231516" watchObservedRunningTime="2026-11-02 01:47:47.954229574 +0000 UTC m=+1384.625482256" Nov 02 01:47:48.609699 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:48.596201 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 02 01:47:48.609699 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:48.596301 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" podUID="25c9e11e-b82c-4e33-81bb-1ccae0588c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 02 01:47:49.213702 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:49.209193 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:49.235647 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:49.228845 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 01:47:49.288701 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:49.286749 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 01:47:49.324080 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:49.324037 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 02 01:47:49.325752 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:49.325718 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:49.375962 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:49.375913 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:49.391830 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:49.391792 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:49.418536 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:49.417812 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:49.491299 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:49.491191 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 02 01:47:49.564696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:49.537937 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:49.564696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:49.555932 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:49.594150 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:49.594111 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 01:47:49.750947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:49.750848 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:50.259697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.231782 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 02 01:47:50.259697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.232439 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:47:50.259697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.233488 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 01:47:50.259697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.233811 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:47:50.295958 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.294079 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:50.295958 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.294463 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:47:50.332696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.331774 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:50.332696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.332169 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 02 01:47:50.332696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.332467 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:47:50.332696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.332527 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:47:50.338692 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.334901 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:50.404697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.387906 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:50.404697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.388315 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:47:50.473694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.472489 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:50.525758 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.491988 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:47:50.526989 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.519291 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:50.527458 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.519331 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": context deadline exceeded" Nov 02 01:47:50.578692 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.527633 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:47:50.578692 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.527786 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:47:50.578692 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.539744 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:50.578692 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.540020 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:47:50.581439 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.581403 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 02 01:47:50.585589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.584006 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:47:50.586299 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.586274 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" podUID="25c9e11e-b82c-4e33-81bb-1ccae0588c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:50.722696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.586200 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:47:50.737107 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.598207 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:50.770733 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.676809 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:47:50.771353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.771311 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" podUID="33422040-a460-438d-a99b-65df8511eeb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:50.806806 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:50.803312 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:47:51.380696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:51.379811 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 01:47:51.380696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:51.380202 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 02 01:47:51.380696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:51.380473 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Nov 02 01:47:51.381012 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:51.380801 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:47:51.475695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:51.467388 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 02 01:47:51.475695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:51.471758 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:51.475695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:51.472961 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:51.475695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:51.473259 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 01:47:51.476046 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:51.475734 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 01:47:51.538817 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:51.538769 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:51.575414 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:51.539839 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:51.601703 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:51.572778 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 02 01:47:51.660756 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:51.596635 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:51.782890 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:51.782845 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:51.977125 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:51.977020 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:47:51.977401 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:51.977365 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" podUID="25c9e11e-b82c-4e33-81bb-1ccae0588c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:52.275775 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:52.255655 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:52.764797 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:52.444093 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Nov 02 01:47:52.764797 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:52.444594 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 01:47:52.764797 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:52.444810 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 02 01:47:52.764797 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:52.602047 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:52.764797 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:52.676491 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 02 01:47:52.764797 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:52.678568 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": context deadline exceeded" Nov 02 01:47:52.764797 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:47:52.705995 5044 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.237s" Nov 02 01:47:52.764797 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:52.734924 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 02 01:47:52.815698 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:52.792355 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:52.815698 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:52.793940 5044 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-kd7nc container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.29:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:47:52.815698 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:52.794011 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-kd7nc" podUID="d75a9f61-49b8-4bf0-b3e5-4200e3ec7332" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.29:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:53.032138 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:52.943795 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 01:47:53.055446 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.055392 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:53.147801 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.122050 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 02 01:47:53.147801 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.125211 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 01:47:53.286733 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.286615 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:53.443983 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.443927 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:53.477738 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.471005 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:47:53.477983 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.477959 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" podUID="25c9e11e-b82c-4e33-81bb-1ccae0588c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:53.663753 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.483018 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf"] Nov 02 01:47:53.676657 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.676604 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded" Nov 02 01:47:53.707163 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.703773 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:53.737375 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.737100 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:47:53.737375 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.737148 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:47:53.737375 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.737169 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf"] Nov 02 01:47:53.738829 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.738221 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf" Nov 02 01:47:53.762810 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.762773 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:47:53.786652 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.784970 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:47:53.786652 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.785024 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:47:53.786652 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.785053 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:47:53.786652 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.785084 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:47:53.791083 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.791062 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:47:53.805696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.800756 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:47:53.846839 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.846817 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:47:53.890731 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.890708 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:47:53.890889 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.890757 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:47:53.890889 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.890789 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:47:53.890889 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.890817 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:47:53.893910 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.893889 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:47:53.894248 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.894230 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\"" Nov 02 01:47:53.896034 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.895999 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kkl9\" (UniqueName: \"kubernetes.io/projected/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-kube-api-access-2kkl9\") pod \"maas-api-revoke-sub-c0ab93b6-mj2zf\" (UID: \"e040acd1-c6d0-43bd-90fb-29bdc08f3a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf" Nov 02 01:47:53.896308 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.896283 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ab93b6-mj2zf\" (UID: \"e040acd1-c6d0-43bd-90fb-29bdc08f3a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf" Nov 02 01:47:53.896639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:53.896615 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ab93b6-mj2zf\" (UID: \"e040acd1-c6d0-43bd-90fb-29bdc08f3a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf" Nov 02 01:47:54.000497 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:54.000375 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kkl9\" (UniqueName: \"kubernetes.io/projected/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-kube-api-access-2kkl9\") pod \"maas-api-revoke-sub-c0ab93b6-mj2zf\" (UID: \"e040acd1-c6d0-43bd-90fb-29bdc08f3a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf" Nov 02 01:47:54.000497 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:54.000471 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ab93b6-mj2zf\" (UID: \"e040acd1-c6d0-43bd-90fb-29bdc08f3a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf" Nov 02 01:47:54.000762 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:54.000744 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ab93b6-mj2zf\" (UID: \"e040acd1-c6d0-43bd-90fb-29bdc08f3a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf" Nov 02 01:47:54.003173 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:54.003130 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ab93b6-mj2zf\" (UID: \"e040acd1-c6d0-43bd-90fb-29bdc08f3a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf" Nov 02 01:47:54.016694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:54.016647 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ab93b6-mj2zf\" (UID: \"e040acd1-c6d0-43bd-90fb-29bdc08f3a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf" Nov 02 01:47:54.073780 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:54.073584 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kkl9\" (UniqueName: \"kubernetes.io/projected/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-kube-api-access-2kkl9\") pod \"maas-api-revoke-sub-c0ab93b6-mj2zf\" (UID: \"e040acd1-c6d0-43bd-90fb-29bdc08f3a3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf" Nov 02 01:47:54.097907 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:54.097857 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:47:54.178363 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:47:54.178321 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:47:54.188546 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:54.188524 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf" Nov 02 01:47:54.326549 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:54.326517 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:54.328191 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:54.327735 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:54.328899 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:54.328874 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:47:54.610543 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:54.610061 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf"] Nov 02 01:47:55.609801 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:55.609760 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf" event={"ID":"e040acd1-c6d0-43bd-90fb-29bdc08f3a3a","Type":"ContainerStarted","Data":"8e926e2b9a3d686c91cc707f8d22553cb53479e5203ae105f3089448f8ecf582"} Nov 02 01:47:55.610291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:55.609826 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf" event={"ID":"e040acd1-c6d0-43bd-90fb-29bdc08f3a3a","Type":"ContainerStarted","Data":"a812628c57cad3041d95ad36e909867e61a50278153904ac261acaf1ebb025d5"} Nov 02 01:47:56.616878 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:56.616651 5044 generic.go:356] "Generic (PLEG): container finished" podID="e040acd1-c6d0-43bd-90fb-29bdc08f3a3a" containerID="8e926e2b9a3d686c91cc707f8d22553cb53479e5203ae105f3089448f8ecf582" exitCode=0 Nov 02 01:47:56.616878 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:56.616814 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf" event={"ID":"e040acd1-c6d0-43bd-90fb-29bdc08f3a3a","Type":"ContainerDied","Data":"8e926e2b9a3d686c91cc707f8d22553cb53479e5203ae105f3089448f8ecf582"} Nov 02 01:47:56.619773 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:56.619729 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf" podStartSLOduration=4.619716651 podStartE2EDuration="4.619716651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:47:56.616872421 +0000 UTC m=+1393.288125103" watchObservedRunningTime="2026-11-02 01:47:56.619716651 +0000 UTC m=+1393.290969332" Nov 02 01:47:57.821871 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:57.821838 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf" Nov 02 01:47:57.880977 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:57.880889 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-maas-api-openshift-ca\") pod \"e040acd1-c6d0-43bd-90fb-29bdc08f3a3a\" (UID: \"e040acd1-c6d0-43bd-90fb-29bdc08f3a3a\") " Nov 02 01:47:57.880977 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:57.880946 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-maas-api-xks-ca\") pod \"e040acd1-c6d0-43bd-90fb-29bdc08f3a3a\" (UID: \"e040acd1-c6d0-43bd-90fb-29bdc08f3a3a\") " Nov 02 01:47:57.881213 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:57.881195 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-kube-api-access-2kkl9\" (UniqueName: \"kubernetes.io/projected/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-kube-api-access-2kkl9\") pod \"e040acd1-c6d0-43bd-90fb-29bdc08f3a3a\" (UID: \"e040acd1-c6d0-43bd-90fb-29bdc08f3a3a\") " Nov 02 01:47:57.881364 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:57.881331 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-maas-api-openshift-ca" pod "e040acd1-c6d0-43bd-90fb-29bdc08f3a3a" (UID: "e040acd1-c6d0-43bd-90fb-29bdc08f3a3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:47:57.881821 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:57.881799 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-maas-api-openshift-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:47:57.883209 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:57.883171 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-maas-api-xks-ca" pod "e040acd1-c6d0-43bd-90fb-29bdc08f3a3a" (UID: "e040acd1-c6d0-43bd-90fb-29bdc08f3a3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:47:57.883581 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:57.883558 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-kube-api-access-2kkl9" pod "e040acd1-c6d0-43bd-90fb-29bdc08f3a3a" (UID: "e040acd1-c6d0-43bd-90fb-29bdc08f3a3a"). InnerVolumeSpecName "kube-api-access-2kkl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:47:57.983282 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:57.983246 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-maas-api-xks-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:47:57.983282 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:57.983285 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kkl9\" (UniqueName: \"kubernetes.io/projected/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a-kube-api-access-2kkl9\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:47:58.632970 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.632940 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf" Nov 02 01:47:58.633139 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.632968 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf" event={"ID":"e040acd1-c6d0-43bd-90fb-29bdc08f3a3a","Type":"ContainerDied","Data":"a812628c57cad3041d95ad36e909867e61a50278153904ac261acaf1ebb025d5"} Nov 02 01:47:58.633139 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.633015 5044 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a812628c57cad3041d95ad36e909867e61a50278153904ac261acaf1ebb025d5" Nov 02 01:47:58.716092 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.716058 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc"] Nov 02 01:47:58.716732 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.716716 5044 cpu_manager.go:414] "containerMap: removing container" podUID="e040acd1-c6d0-43bd-90fb-29bdc08f3a3a" containerName="revoke-keys" Nov 02 01:47:58.716786 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.716732 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e040acd1-c6d0-43bd-90fb-29bdc08f3a3a" containerName="revoke-keys" Nov 02 01:47:58.716872 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.716863 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="e040acd1-c6d0-43bd-90fb-29bdc08f3a3a" containerName="revoke-keys" Nov 02 01:47:58.720776 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.720753 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc" Nov 02 01:47:58.775711 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.773170 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc"] Nov 02 01:47:58.782506 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.781829 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\"" Nov 02 01:47:58.782506 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.781961 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:47:58.796014 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.795982 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c631f891-1a57-469f-ba37-00f34847792e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ad04b64-stqmc\" (UID: \"c631f891-1a57-469f-ba37-00f34847792e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc" Nov 02 01:47:58.796288 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.796266 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c631f891-1a57-469f-ba37-00f34847792e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ad04b64-stqmc\" (UID: \"c631f891-1a57-469f-ba37-00f34847792e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc" Nov 02 01:47:58.796371 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.796315 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr8p7\" (UniqueName: \"kubernetes.io/projected/c631f891-1a57-469f-ba37-00f34847792e-kube-api-access-dr8p7\") pod \"maas-api-revoke-sub-2ad04b64-stqmc\" (UID: \"c631f891-1a57-469f-ba37-00f34847792e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc" Nov 02 01:47:58.893936 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.893843 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:47:58.898343 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.898316 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c631f891-1a57-469f-ba37-00f34847792e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ad04b64-stqmc\" (UID: \"c631f891-1a57-469f-ba37-00f34847792e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc" Nov 02 01:47:58.898450 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.898367 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr8p7\" (UniqueName: \"kubernetes.io/projected/c631f891-1a57-469f-ba37-00f34847792e-kube-api-access-dr8p7\") pod \"maas-api-revoke-sub-2ad04b64-stqmc\" (UID: \"c631f891-1a57-469f-ba37-00f34847792e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc" Nov 02 01:47:58.898508 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.898464 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c631f891-1a57-469f-ba37-00f34847792e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ad04b64-stqmc\" (UID: \"c631f891-1a57-469f-ba37-00f34847792e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc" Nov 02 01:47:58.899180 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.899153 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c631f891-1a57-469f-ba37-00f34847792e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ad04b64-stqmc\" (UID: \"c631f891-1a57-469f-ba37-00f34847792e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc" Nov 02 01:47:58.901107 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.901080 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c631f891-1a57-469f-ba37-00f34847792e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ad04b64-stqmc\" (UID: \"c631f891-1a57-469f-ba37-00f34847792e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc" Nov 02 01:47:58.961179 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:58.961145 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr8p7\" (UniqueName: \"kubernetes.io/projected/c631f891-1a57-469f-ba37-00f34847792e-kube-api-access-dr8p7\") pod \"maas-api-revoke-sub-2ad04b64-stqmc\" (UID: \"c631f891-1a57-469f-ba37-00f34847792e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc" Nov 02 01:47:59.035129 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:59.035094 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc" Nov 02 01:47:59.426869 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:59.426838 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc"] Nov 02 01:47:59.427353 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:47:59.427322 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc631f891_1a57_469f_ba37_00f34847792e.slice/crio-c6e3e485a171901ea2c1feaf74b5fc488c069d5a9097587d50f47e444af795d8 WatchSource:0}: Error finding container c6e3e485a171901ea2c1feaf74b5fc488c069d5a9097587d50f47e444af795d8: Status 404 returned error can't find the container with id c6e3e485a171901ea2c1feaf74b5fc488c069d5a9097587d50f47e444af795d8 Nov 02 01:47:59.640943 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:47:59.640914 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc" event={"ID":"c631f891-1a57-469f-ba37-00f34847792e","Type":"ContainerStarted","Data":"c6e3e485a171901ea2c1feaf74b5fc488c069d5a9097587d50f47e444af795d8"} Nov 02 01:48:00.649035 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:00.648936 5044 generic.go:356] "Generic (PLEG): container finished" podID="c631f891-1a57-469f-ba37-00f34847792e" containerID="0a5238cf38d8ce14ca021378490b2a9eb0456f9e5fb5ede7ebb05f38728aee47" exitCode=0 Nov 02 01:48:00.649423 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:00.649013 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc" event={"ID":"c631f891-1a57-469f-ba37-00f34847792e","Type":"ContainerDied","Data":"0a5238cf38d8ce14ca021378490b2a9eb0456f9e5fb5ede7ebb05f38728aee47"} Nov 02 01:48:01.838164 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:01.838135 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc" Nov 02 01:48:01.938991 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:01.938920 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c631f891-1a57-469f-ba37-00f34847792e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c631f891-1a57-469f-ba37-00f34847792e-maas-api-xks-ca\") pod \"c631f891-1a57-469f-ba37-00f34847792e\" (UID: \"c631f891-1a57-469f-ba37-00f34847792e\") " Nov 02 01:48:01.938991 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:01.938988 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c631f891-1a57-469f-ba37-00f34847792e-kube-api-access-dr8p7\" (UniqueName: \"kubernetes.io/projected/c631f891-1a57-469f-ba37-00f34847792e-kube-api-access-dr8p7\") pod \"c631f891-1a57-469f-ba37-00f34847792e\" (UID: \"c631f891-1a57-469f-ba37-00f34847792e\") " Nov 02 01:48:01.939170 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:01.939049 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c631f891-1a57-469f-ba37-00f34847792e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c631f891-1a57-469f-ba37-00f34847792e-maas-api-openshift-ca\") pod \"c631f891-1a57-469f-ba37-00f34847792e\" (UID: \"c631f891-1a57-469f-ba37-00f34847792e\") " Nov 02 01:48:01.939423 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:01.939393 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c631f891-1a57-469f-ba37-00f34847792e-maas-api-openshift-ca" pod "c631f891-1a57-469f-ba37-00f34847792e" (UID: "c631f891-1a57-469f-ba37-00f34847792e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:48:01.941062 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:01.941034 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c631f891-1a57-469f-ba37-00f34847792e-maas-api-xks-ca" pod "c631f891-1a57-469f-ba37-00f34847792e" (UID: "c631f891-1a57-469f-ba37-00f34847792e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:48:01.941062 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:01.941041 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c631f891-1a57-469f-ba37-00f34847792e-kube-api-access-dr8p7" pod "c631f891-1a57-469f-ba37-00f34847792e" (UID: "c631f891-1a57-469f-ba37-00f34847792e"). InnerVolumeSpecName "kube-api-access-dr8p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:02.041341 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:02.041305 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c631f891-1a57-469f-ba37-00f34847792e-maas-api-xks-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:48:02.041341 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:02.041340 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr8p7\" (UniqueName: \"kubernetes.io/projected/c631f891-1a57-469f-ba37-00f34847792e-kube-api-access-dr8p7\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:48:02.041341 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:02.041351 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c631f891-1a57-469f-ba37-00f34847792e-maas-api-openshift-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:48:02.664733 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:02.664666 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc" Nov 02 01:48:02.664954 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:02.664760 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc" event={"ID":"c631f891-1a57-469f-ba37-00f34847792e","Type":"ContainerDied","Data":"c6e3e485a171901ea2c1feaf74b5fc488c069d5a9097587d50f47e444af795d8"} Nov 02 01:48:02.664954 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:02.664818 5044 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6e3e485a171901ea2c1feaf74b5fc488c069d5a9097587d50f47e444af795d8" Nov 02 01:48:03.722089 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:03.722056 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:08.604900 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:08.604868 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:09.978652 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:09.978609 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz"] Nov 02 01:48:09.979447 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:09.979425 5044 cpu_manager.go:414] "containerMap: removing container" podUID="c631f891-1a57-469f-ba37-00f34847792e" containerName="revoke-keys" Nov 02 01:48:09.979564 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:09.979447 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c631f891-1a57-469f-ba37-00f34847792e" containerName="revoke-keys" Nov 02 01:48:09.979655 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:09.979642 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="c631f891-1a57-469f-ba37-00f34847792e" containerName="revoke-keys" Nov 02 01:48:09.983298 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:09.983278 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.091531 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.091501 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz"] Nov 02 01:48:10.146459 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.146424 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sldl\" (UniqueName: \"kubernetes.io/projected/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-kube-api-access-9sldl\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.146459 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.146462 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.146653 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.146481 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.146736 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.146650 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.146778 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.146767 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.146872 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.146856 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.146973 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.146946 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.147031 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.146989 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.147094 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.147076 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.248588 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.248498 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sldl\" (UniqueName: \"kubernetes.io/projected/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-kube-api-access-9sldl\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.248588 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.248558 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.248588 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.248584 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.248900 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.248638 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.248900 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.248773 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.248900 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.248843 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.249055 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.248915 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.249055 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.248950 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.249055 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.249008 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.249298 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.249163 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-workload-certs\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.249298 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.249223 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-workload-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.249422 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.249300 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-credential-socket\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.249513 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.249490 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-data\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.249584 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.249567 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istiod-ca-cert\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.251485 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.251462 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-envoy\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.251755 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.251734 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-podinfo\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.310838 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.310810 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sldl\" (UniqueName: \"kubernetes.io/projected/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-kube-api-access-9sldl\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.310838 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.310825 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-token\") pod \"e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.437137 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.437099 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559"] Nov 02 01:48:10.442366 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.442348 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.517897 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.517864 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559"] Nov 02 01:48:10.552629 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.552597 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9008e53-2fc7-44ab-b841-e4eea224fded-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.552629 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.552631 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.552827 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.552661 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsk4w\" (UniqueName: \"kubernetes.io/projected/f9008e53-2fc7-44ab-b841-e4eea224fded-kube-api-access-lsk4w\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.552827 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.552781 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.552895 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.552871 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.552970 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.552959 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.553084 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.553072 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.553119 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.553093 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.553200 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.553183 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.599223 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.599191 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:10.655308 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.655275 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.655497 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.655372 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.655497 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.655403 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.655619 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.655502 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.655619 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.655554 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9008e53-2fc7-44ab-b841-e4eea224fded-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.655619 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.655582 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.655848 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.655628 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsk4w\" (UniqueName: \"kubernetes.io/projected/f9008e53-2fc7-44ab-b841-e4eea224fded-kube-api-access-lsk4w\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.655848 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.655661 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.655848 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.655765 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.656295 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.656228 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-credential-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.656747 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.656720 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-workload-certs\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.657214 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.657182 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-workload-socket\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.657736 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.657717 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-data\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.658873 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.658849 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9008e53-2fc7-44ab-b841-e4eea224fded-istiod-ca-cert\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.659646 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.659625 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-envoy\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.660511 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.660478 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-podinfo\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.727589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.727557 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-token\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.727774 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.727595 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsk4w\" (UniqueName: \"kubernetes.io/projected/f9008e53-2fc7-44ab-b841-e4eea224fded-kube-api-access-lsk4w\") pod \"e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.757592 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.757561 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:10.972402 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.972368 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k"] Nov 02 01:48:10.978055 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.978031 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:10.989659 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.989634 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284"] Nov 02 01:48:10.994092 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:10.994077 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.041581 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.041507 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k"] Nov 02 01:48:11.064883 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.064850 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.065026 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.064958 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.065026 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.064996 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.065113 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.065035 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.065235 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.065205 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wggb8\" (UniqueName: \"kubernetes.io/projected/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-kube-api-access-wggb8\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.065339 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.065318 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.065401 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.065384 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.065460 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.065425 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.065647 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.065616 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.068191 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.068169 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284"] Nov 02 01:48:11.098630 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.098594 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz"] Nov 02 01:48:11.102536 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:48:11.102504 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddace65e4_e8b0_4a4b_84cf_b6bc1e0e7258.slice/crio-e9b00f89e1406221804ab910363fae17288425b92775d995d2d8bd35e0c3c1cf WatchSource:0}: Error finding container e9b00f89e1406221804ab910363fae17288425b92775d995d2d8bd35e0c3c1cf: Status 404 returned error can't find the container with id e9b00f89e1406221804ab910363fae17288425b92775d995d2d8bd35e0c3c1cf Nov 02 01:48:11.108524 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.108481 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:11.108635 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.108570 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:11.108635 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.108617 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:11.166998 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.166972 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.167128 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.167055 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.167210 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.167138 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.167210 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.167175 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.167324 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.167244 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.167324 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.167276 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.167428 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.167348 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.167428 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.167389 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.167428 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.167420 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.167643 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.167617 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.167799 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.167778 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.167866 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.167834 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wggb8\" (UniqueName: \"kubernetes.io/projected/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-kube-api-access-wggb8\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.167970 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.167951 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.168036 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.167996 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.168094 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.168028 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.168094 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.167954 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.168094 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.168060 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.168225 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.168150 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7365ed-68db-449b-9ef1-047cc14f55c3-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.168272 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.168220 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.168324 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.168263 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bztsj\" (UniqueName: \"kubernetes.io/projected/2d7365ed-68db-449b-9ef1-047cc14f55c3-kube-api-access-bztsj\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.168324 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.168300 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.169331 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.168553 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.169331 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.168642 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.170386 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.170364 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.171135 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.171061 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.174034 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:48:11.174014 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9008e53_2fc7_44ab_b841_e4eea224fded.slice/crio-7ba41ef97db99158553eea634297cbba807efdcf3d3e331c60d75ea8d9527065 WatchSource:0}: Error finding container 7ba41ef97db99158553eea634297cbba807efdcf3d3e331c60d75ea8d9527065: Status 404 returned error can't find the container with id 7ba41ef97db99158553eea634297cbba807efdcf3d3e331c60d75ea8d9527065 Nov 02 01:48:11.175257 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.175236 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559"] Nov 02 01:48:11.176963 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.176937 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:11.177024 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.176997 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:11.177064 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.177030 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:11.229445 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.229399 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wggb8\" (UniqueName: \"kubernetes.io/projected/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-kube-api-access-wggb8\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.230650 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.230601 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.276466 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.269486 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.276466 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.269553 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7365ed-68db-449b-9ef1-047cc14f55c3-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.276466 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.269579 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bztsj\" (UniqueName: \"kubernetes.io/projected/2d7365ed-68db-449b-9ef1-047cc14f55c3-kube-api-access-bztsj\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.276466 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.269623 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.276466 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.269704 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.276466 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.269770 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.276466 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.269798 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.276466 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.269813 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.276466 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.269894 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.276466 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.270246 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-data\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.276466 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.270746 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-workload-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.276466 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.271010 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-workload-certs\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.276466 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.271490 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-credential-socket\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.276466 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.271572 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7365ed-68db-449b-9ef1-047cc14f55c3-istiod-ca-cert\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.276466 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.273614 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-podinfo\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.276466 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.273931 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-envoy\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.296393 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.293460 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:11.337347 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.337206 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bztsj\" (UniqueName: \"kubernetes.io/projected/2d7365ed-68db-449b-9ef1-047cc14f55c3-kube-api-access-bztsj\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.337973 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.337954 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-token\") pod \"e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.607049 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.606953 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:11.712209 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.711914 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k"] Nov 02 01:48:11.730950 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.729636 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:11.730950 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.729753 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:11.730950 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.729798 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:11.758187 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.758143 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" event={"ID":"f9008e53-2fc7-44ab-b841-e4eea224fded","Type":"ContainerStarted","Data":"89ef6add7905cab77444f3b6a5a42d0e02c466bb33e76e4b7fc9a80138f82426"} Nov 02 01:48:11.758466 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.758437 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" event={"ID":"f9008e53-2fc7-44ab-b841-e4eea224fded","Type":"ContainerStarted","Data":"7ba41ef97db99158553eea634297cbba807efdcf3d3e331c60d75ea8d9527065"} Nov 02 01:48:11.786301 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.786267 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:11.827691 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.827332 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" event={"ID":"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258","Type":"ContainerStarted","Data":"cadf401c0c25ae7a51a1b148632572bcb9e56bdbfcee8d6deaffe37721841555"} Nov 02 01:48:11.827691 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.827397 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" event={"ID":"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258","Type":"ContainerStarted","Data":"e9b00f89e1406221804ab910363fae17288425b92775d995d2d8bd35e0c3c1cf"} Nov 02 01:48:11.831883 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.830017 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" event={"ID":"1dbf2ea9-cb23-4fa2-b246-58ca09066b74","Type":"ContainerStarted","Data":"9bfbbea2ae9dc49c2a85d3a5b978f56dcb0ba28052abc5d6b4b93c0d84266987"} Nov 02 01:48:11.922692 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:11.917143 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" podStartSLOduration=1.91712236 podStartE2EDuration="1.91712236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:11.915660862 +0000 UTC m=+1408.586913543" watchObservedRunningTime="2026-11-02 01:48:11.91712236 +0000 UTC m=+1408.588375045" Nov 02 01:48:12.031219 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:12.031185 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284"] Nov 02 01:48:12.033498 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:48:12.031601 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7365ed_68db_449b_9ef1_047cc14f55c3.slice/crio-e71951af94d091c41f4502789bcaca7f05984135cabfc071b42892894103ed0c WatchSource:0}: Error finding container e71951af94d091c41f4502789bcaca7f05984135cabfc071b42892894103ed0c: Status 404 returned error can't find the container with id e71951af94d091c41f4502789bcaca7f05984135cabfc071b42892894103ed0c Nov 02 01:48:12.036427 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:12.036398 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:12.036530 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:12.036465 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:12.036530 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:12.036494 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:12.054709 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:12.052065 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7"] Nov 02 01:48:12.059099 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:12.059063 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-634cc270-z88h7"] Nov 02 01:48:12.601695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:12.601056 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:12.620694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:12.616805 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 01:48:12.620694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:12.616865 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" podUID="dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 01:48:12.778767 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:12.778735 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 02 01:48:12.779008 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:12.778980 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" podUID="f9008e53-2fc7-44ab-b841-e4eea224fded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 02 01:48:13.051502 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:13.051449 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" event={"ID":"1dbf2ea9-cb23-4fa2-b246-58ca09066b74","Type":"ContainerStarted","Data":"d907db85be7e47d9345985af951de423ee97fdec394c1640192ca41bdaffa41f"} Nov 02 01:48:13.118767 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:13.118729 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" event={"ID":"2d7365ed-68db-449b-9ef1-047cc14f55c3","Type":"ContainerStarted","Data":"86412078dfd576c857e7ae73bc01d9684ce0c333c38094343c81e3ce1ea6ea23"} Nov 02 01:48:13.119048 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:13.119027 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" event={"ID":"2d7365ed-68db-449b-9ef1-047cc14f55c3","Type":"ContainerStarted","Data":"e71951af94d091c41f4502789bcaca7f05984135cabfc071b42892894103ed0c"} Nov 02 01:48:13.244697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:13.223579 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" podStartSLOduration=4.223557022 podStartE2EDuration="4.223557022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:12.102409626 +0000 UTC m=+1408.773662308" watchObservedRunningTime="2026-11-02 01:48:13.223557022 +0000 UTC m=+1409.894809706" Nov 02 01:48:13.246980 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:13.246916 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" podStartSLOduration=3.246899884 podStartE2EDuration="3.246899884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:13.218270912 +0000 UTC m=+1409.889523585" watchObservedRunningTime="2026-11-02 01:48:13.246899884 +0000 UTC m=+1409.918152568" Nov 02 01:48:13.305794 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:13.305699 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:13.309692 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:13.306426 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 01:48:13.309692 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:13.306490 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" podUID="1dbf2ea9-cb23-4fa2-b246-58ca09066b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 01:48:13.373922 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:13.373856 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" podStartSLOduration=3.373823586 podStartE2EDuration="3.373823586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:13.368564749 +0000 UTC m=+1410.039817431" watchObservedRunningTime="2026-11-02 01:48:13.373823586 +0000 UTC m=+1410.045076267" Nov 02 01:48:13.594747 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:13.585419 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6bcae83-1781-4dd1-bca6-fcd63d380dfa" path="/var/lib/kubelet/pods/c6bcae83-1781-4dd1-bca6-fcd63d380dfa/volumes" Nov 02 01:48:13.603480 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:13.603040 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 01:48:13.603480 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:13.603100 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" podUID="dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 01:48:13.608041 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:13.607978 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:13.616892 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:13.616774 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:48:13.616892 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:13.616829 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" podUID="2d7365ed-68db-449b-9ef1-047cc14f55c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 01:48:13.763326 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:13.763287 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 02 01:48:13.763610 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:13.763580 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" podUID="f9008e53-2fc7-44ab-b841-e4eea224fded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 02 01:48:13.858784 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:13.858700 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:14.296694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.296075 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 01:48:14.296694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.296137 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" podUID="1dbf2ea9-cb23-4fa2-b246-58ca09066b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 01:48:14.468701 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.460200 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Nov 02 01:48:14.515695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.511110 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 01:48:14.536732 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.532759 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:14.560754 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.553352 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:14.608703 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.607834 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:48:14.608703 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.607892 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" podUID="2d7365ed-68db-449b-9ef1-047cc14f55c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 01:48:14.608703 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.608372 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 01:48:14.611806 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.609317 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 01:48:14.611806 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.609356 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" podUID="dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 01:48:14.667312 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.667272 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:14.687855 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.687794 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:14.768728 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.767601 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:14.768728 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.768021 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Nov 02 01:48:14.768728 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.768080 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 02 01:48:14.768728 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.768119 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 02 01:48:14.773702 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.771797 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 02 01:48:14.773702 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.771869 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" podUID="f9008e53-2fc7-44ab-b841-e4eea224fded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 02 01:48:14.905769 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.902842 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:14.988720 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.952396 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:14.988720 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:14.956984 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 02 01:48:15.544753 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.531724 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:15.544753 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.541281 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:15.553704 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.550451 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 01:48:15.553704 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.550505 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" podUID="1dbf2ea9-cb23-4fa2-b246-58ca09066b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 01:48:15.588314 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.554449 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:15.610892 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.610848 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:15.638829 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.620813 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:48:15.638829 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.620888 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" podUID="2d7365ed-68db-449b-9ef1-047cc14f55c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 01:48:15.638829 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.621439 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 01:48:15.638829 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.621477 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" podUID="dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 01:48:15.678809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.678761 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:15.704820 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.690376 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:15.773439 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.773386 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:15.774116 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.774082 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:15.774344 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.774320 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": context deadline exceeded" Nov 02 01:48:15.774660 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.774637 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 02 01:48:15.798146 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.798071 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:48:15.798402 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.798386 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:48:15.798540 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.798527 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:48:15.798682 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.798655 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:48:15.798836 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.798822 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:48:15.798974 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.798962 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:48:15.799096 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.799084 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:48:15.816695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.802750 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:48:15.816695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.802805 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:48:15.816695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.810500 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 02 01:48:15.816695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.810542 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" podUID="f9008e53-2fc7-44ab-b841-e4eea224fded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 02 01:48:15.854880 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.854842 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": context deadline exceeded" Nov 02 01:48:15.917299 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.916786 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:15.918101 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.918078 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:48:15.992695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.971861 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:15.992695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.973292 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 01:48:15.992695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.990036 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:48:15.992695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:15.990126 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:48:16.307696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.303623 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 01:48:16.419698 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.419630 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" podUID="1dbf2ea9-cb23-4fa2-b246-58ca09066b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 01:48:16.470993 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.470950 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:16.500774 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.471296 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:16.531761 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.525095 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:16.531761 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.530753 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:16.541689 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.532057 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:16.554063 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.542738 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 01:48:16.561904 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.561821 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 02 01:48:16.679694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.609604 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 01:48:16.679694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.609662 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" podUID="dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 01:48:16.679694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.609951 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:48:16.679694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.609984 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" podUID="2d7365ed-68db-449b-9ef1-047cc14f55c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 01:48:16.679694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.652822 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:16.692702 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.686584 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:16.693267 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.690915 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:16.774864 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.774798 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 02 01:48:16.814855 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.814749 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" podUID="f9008e53-2fc7-44ab-b841-e4eea224fded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 02 01:48:16.853693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.781850 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 02 01:48:16.878696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.781873 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Nov 02 01:48:16.878696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.781893 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": context deadline exceeded" Nov 02 01:48:16.878696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.781812 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:16.939917 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.939849 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 02 01:48:16.968989 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:16.967988 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:48:17.023518 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.021107 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:17.023518 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.021898 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 01:48:17.218369 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.218281 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:48:17.404701 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.391294 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 01:48:17.404986 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.404954 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" podUID="1dbf2ea9-cb23-4fa2-b246-58ca09066b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 01:48:17.405138 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.401030 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:48:17.568092 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.563346 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 01:48:17.568092 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.563636 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:17.580761 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.572825 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:17.600743 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.600703 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 01:48:17.604225 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.600974 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" podUID="dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 01:48:17.630745 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.619688 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:48:17.630745 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.619741 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" podUID="2d7365ed-68db-449b-9ef1-047cc14f55c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 01:48:17.663696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.655172 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:17.719701 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.697353 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:17.900756 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.886789 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:17.900756 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.887151 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:17.900756 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.887468 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:48:17.900756 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.892960 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 02 01:48:17.900756 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.893000 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" podUID="f9008e53-2fc7-44ab-b841-e4eea224fded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 02 01:48:17.986740 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:17.963755 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:18.057706 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.053871 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:18.057706 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.054240 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 02 01:48:18.282763 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.218330 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Nov 02 01:48:18.730484 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.277839 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 01:48:18.731717 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.317933 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 01:48:18.731906 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.731877 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" podUID="1dbf2ea9-cb23-4fa2-b246-58ca09066b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 01:48:18.732044 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.581660 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:18.733304 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.443296 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:18.734050 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.667406 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 01:48:18.739758 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.739729 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" podUID="dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 01:48:18.739901 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.598865 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:18.747684 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.598909 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 01:48:18.748913 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.668336 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:48:18.775863 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.775817 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" podUID="2d7365ed-68db-449b-9ef1-047cc14f55c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 01:48:18.776118 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.739178 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:18.778107 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.739215 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:18.783752 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.780588 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 02 01:48:18.783985 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.783945 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" podUID="f9008e53-2fc7-44ab-b841-e4eea224fded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 02 01:48:18.904019 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.903963 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:18.916806 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.916765 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 02 01:48:18.917085 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:18.917058 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Nov 02 01:48:19.100731 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.095377 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 02 01:48:19.107735 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.103617 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 01:48:19.129720 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.123061 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 02 01:48:19.314692 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.309881 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 01:48:19.314692 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.309948 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" podUID="1dbf2ea9-cb23-4fa2-b246-58ca09066b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 01:48:19.642640 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.642602 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:48:19.642926 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.642894 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" podUID="2d7365ed-68db-449b-9ef1-047cc14f55c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 01:48:19.643160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.643141 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 01:48:19.643295 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.643275 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" podUID="dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 01:48:19.750697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.748848 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:19.750697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.750153 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:19.751269 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.750829 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:48:19.800700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.758705 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 01:48:19.800700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.758900 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 01:48:19.800700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.758958 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:19.800700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.759149 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:48:19.800700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.765817 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 02 01:48:19.800700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.765858 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" podUID="f9008e53-2fc7-44ab-b841-e4eea224fded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 02 01:48:19.800700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.792272 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:19.800700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.796738 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 02 01:48:19.826706 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.822755 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:19.929080 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.928987 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:19.953735 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.929299 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:19.954747 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.929337 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:19.955615 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:19.955594 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:48:20.111224 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.111178 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:20.125189 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.125146 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:20.149808 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.149750 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:20.549660 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.547327 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:48:20.549660 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.548545 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:48:20.552353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.552335 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:48:20.570333 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.570312 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:48:20.586866 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.586827 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:48:20.606065 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.606028 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:48:20.664172 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.658117 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:48:20.749804 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.749776 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:48:20.750360 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.675978 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:48:20.750541 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.750514 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" podUID="2d7365ed-68db-449b-9ef1-047cc14f55c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 01:48:20.752704 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.710513 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 01:48:20.753186 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.753156 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" podUID="dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 01:48:20.753319 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.748326 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 01:48:20.753453 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.753432 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" podUID="1dbf2ea9-cb23-4fa2-b246-58ca09066b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 01:48:20.754009 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.753989 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:48:20.755138 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.755106 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:20.765253 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.765234 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:48:20.804861 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.804773 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:20.957014 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.951218 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:48:20.972790 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:20.971061 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:48:21.062727 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:21.061793 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:48:21.257570 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:21.256343 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:21.260586 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:21.258251 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:21.304693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:21.304358 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:48:21.318405 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:21.314174 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 02 01:48:21.318405 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:21.314244 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" podUID="1dbf2ea9-cb23-4fa2-b246-58ca09066b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 02 01:48:21.613397 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:21.610756 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 02 01:48:21.613397 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:21.610812 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" podUID="2d7365ed-68db-449b-9ef1-047cc14f55c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 02 01:48:21.613397 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:21.611065 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:21.613397 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:21.612289 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:21.619964 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:21.619936 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 01:48:21.853346 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:21.853259 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:22.311892 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:22.310536 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:22.317856 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:22.312186 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:22.328232 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:22.327979 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:48:22.619210 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:22.619136 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:22.620051 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:22.619972 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:22.623261 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:22.623238 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:48:23.637197 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:23.637164 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr"] Nov 02 01:48:23.643977 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:23.643954 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr" Nov 02 01:48:23.687020 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:23.686988 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d78e2c-221f-47f2-966f-bd42c211bae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c41b753-v6gxr\" (UID: \"d0d78e2c-221f-47f2-966f-bd42c211bae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr" Nov 02 01:48:23.687219 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:23.687201 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vdlx\" (UniqueName: \"kubernetes.io/projected/d0d78e2c-221f-47f2-966f-bd42c211bae9-kube-api-access-2vdlx\") pod \"maas-api-revoke-sub-8c41b753-v6gxr\" (UID: \"d0d78e2c-221f-47f2-966f-bd42c211bae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr" Nov 02 01:48:23.687297 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:23.687276 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d78e2c-221f-47f2-966f-bd42c211bae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c41b753-v6gxr\" (UID: \"d0d78e2c-221f-47f2-966f-bd42c211bae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr" Nov 02 01:48:23.697294 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:23.697271 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:48:23.697499 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:23.697376 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\"" Nov 02 01:48:23.700167 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:23.700141 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr"] Nov 02 01:48:23.789195 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:23.789158 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d78e2c-221f-47f2-966f-bd42c211bae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c41b753-v6gxr\" (UID: \"d0d78e2c-221f-47f2-966f-bd42c211bae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr" Nov 02 01:48:23.789545 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:23.789510 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vdlx\" (UniqueName: \"kubernetes.io/projected/d0d78e2c-221f-47f2-966f-bd42c211bae9-kube-api-access-2vdlx\") pod \"maas-api-revoke-sub-8c41b753-v6gxr\" (UID: \"d0d78e2c-221f-47f2-966f-bd42c211bae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr" Nov 02 01:48:23.789690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:23.789579 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d78e2c-221f-47f2-966f-bd42c211bae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c41b753-v6gxr\" (UID: \"d0d78e2c-221f-47f2-966f-bd42c211bae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr" Nov 02 01:48:23.790171 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:23.790147 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d78e2c-221f-47f2-966f-bd42c211bae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c41b753-v6gxr\" (UID: \"d0d78e2c-221f-47f2-966f-bd42c211bae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr" Nov 02 01:48:23.792182 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:23.792152 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d78e2c-221f-47f2-966f-bd42c211bae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c41b753-v6gxr\" (UID: \"d0d78e2c-221f-47f2-966f-bd42c211bae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr" Nov 02 01:48:23.855835 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:23.855805 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vdlx\" (UniqueName: \"kubernetes.io/projected/d0d78e2c-221f-47f2-966f-bd42c211bae9-kube-api-access-2vdlx\") pod \"maas-api-revoke-sub-8c41b753-v6gxr\" (UID: \"d0d78e2c-221f-47f2-966f-bd42c211bae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr" Nov 02 01:48:23.958959 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:23.958865 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr" Nov 02 01:48:24.181591 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:48:24.181547 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:48:24.358999 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:48:24.358971 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d78e2c_221f_47f2_966f_bd42c211bae9.slice/crio-07ec8b20de7c689baae3f91d122285f94836a1d0704c0fee134b4b2fddae0716 WatchSource:0}: Error finding container 07ec8b20de7c689baae3f91d122285f94836a1d0704c0fee134b4b2fddae0716: Status 404 returned error can't find the container with id 07ec8b20de7c689baae3f91d122285f94836a1d0704c0fee134b4b2fddae0716 Nov 02 01:48:24.359180 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:24.359156 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr"] Nov 02 01:48:24.782863 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:24.782794 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr" event={"ID":"d0d78e2c-221f-47f2-966f-bd42c211bae9","Type":"ContainerStarted","Data":"97e5795c2a597e4457a9a4b870dd042539975dcb1ded9e759f5977fcd0f9c6d6"} Nov 02 01:48:24.783334 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:24.782914 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr" event={"ID":"d0d78e2c-221f-47f2-966f-bd42c211bae9","Type":"ContainerStarted","Data":"07ec8b20de7c689baae3f91d122285f94836a1d0704c0fee134b4b2fddae0716"} Nov 02 01:48:24.902940 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:24.902889 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr" podStartSLOduration=1.9028765 podStartE2EDuration="1.9028765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:24.89986706 +0000 UTC m=+1421.571119742" watchObservedRunningTime="2026-11-02 01:48:24.9028765 +0000 UTC m=+1421.574129181" Nov 02 01:48:25.289377 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:25.289345 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:25.792616 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:25.792576 5044 generic.go:356] "Generic (PLEG): container finished" podID="d0d78e2c-221f-47f2-966f-bd42c211bae9" containerID="97e5795c2a597e4457a9a4b870dd042539975dcb1ded9e759f5977fcd0f9c6d6" exitCode=0 Nov 02 01:48:25.793156 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:25.792656 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr" event={"ID":"d0d78e2c-221f-47f2-966f-bd42c211bae9","Type":"ContainerDied","Data":"97e5795c2a597e4457a9a4b870dd042539975dcb1ded9e759f5977fcd0f9c6d6"} Nov 02 01:48:26.992207 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:26.992178 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr" Nov 02 01:48:27.056744 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:27.056410 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0d78e2c-221f-47f2-966f-bd42c211bae9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d78e2c-221f-47f2-966f-bd42c211bae9-maas-api-xks-ca\") pod \"d0d78e2c-221f-47f2-966f-bd42c211bae9\" (UID: \"d0d78e2c-221f-47f2-966f-bd42c211bae9\") " Nov 02 01:48:27.056744 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:27.056584 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0d78e2c-221f-47f2-966f-bd42c211bae9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d78e2c-221f-47f2-966f-bd42c211bae9-maas-api-openshift-ca\") pod \"d0d78e2c-221f-47f2-966f-bd42c211bae9\" (UID: \"d0d78e2c-221f-47f2-966f-bd42c211bae9\") " Nov 02 01:48:27.056744 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:27.056691 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d78e2c-221f-47f2-966f-bd42c211bae9-kube-api-access-2vdlx\" (UniqueName: \"kubernetes.io/projected/d0d78e2c-221f-47f2-966f-bd42c211bae9-kube-api-access-2vdlx\") pod \"d0d78e2c-221f-47f2-966f-bd42c211bae9\" (UID: \"d0d78e2c-221f-47f2-966f-bd42c211bae9\") " Nov 02 01:48:27.060688 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:27.058996 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d78e2c-221f-47f2-966f-bd42c211bae9-maas-api-openshift-ca" pod "d0d78e2c-221f-47f2-966f-bd42c211bae9" (UID: "d0d78e2c-221f-47f2-966f-bd42c211bae9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:48:27.064683 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:27.061079 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d78e2c-221f-47f2-966f-bd42c211bae9-kube-api-access-2vdlx" pod "d0d78e2c-221f-47f2-966f-bd42c211bae9" (UID: "d0d78e2c-221f-47f2-966f-bd42c211bae9"). InnerVolumeSpecName "kube-api-access-2vdlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:27.101251 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:27.098349 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d78e2c-221f-47f2-966f-bd42c211bae9-maas-api-xks-ca" pod "d0d78e2c-221f-47f2-966f-bd42c211bae9" (UID: "d0d78e2c-221f-47f2-966f-bd42c211bae9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:48:27.175710 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:27.161984 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vdlx\" (UniqueName: \"kubernetes.io/projected/d0d78e2c-221f-47f2-966f-bd42c211bae9-kube-api-access-2vdlx\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:48:27.175710 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:27.162025 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d78e2c-221f-47f2-966f-bd42c211bae9-maas-api-xks-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:48:27.175710 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:27.162041 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d78e2c-221f-47f2-966f-bd42c211bae9-maas-api-openshift-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:48:27.987919 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:27.986625 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr" event={"ID":"d0d78e2c-221f-47f2-966f-bd42c211bae9","Type":"ContainerDied","Data":"07ec8b20de7c689baae3f91d122285f94836a1d0704c0fee134b4b2fddae0716"} Nov 02 01:48:27.987919 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:27.986717 5044 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07ec8b20de7c689baae3f91d122285f94836a1d0704c0fee134b4b2fddae0716" Nov 02 01:48:27.987919 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:27.986792 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr" Nov 02 01:48:28.551962 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:28.551912 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:28.556269 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:28.556231 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 01:48:28.615700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:28.564761 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 01:48:28.615700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:28.582684 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 02 01:48:28.615700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:28.611852 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": context deadline exceeded" Nov 02 01:48:28.728767 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:28.702048 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:28.729800 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:28.729752 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:28.796967 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:28.796918 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:28.914755 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:28.913764 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:28.942694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:28.942017 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:28.964884 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:28.964835 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:29.042696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:29.023925 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:29.057587 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:29.057323 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:29.558825 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:29.558772 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:29.563359 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:29.563334 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:48:29.596049 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:29.560932 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:29.596049 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:29.564296 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:48:29.596049 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:29.574755 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 01:48:29.596049 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:29.574955 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:48:29.596049 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:29.584856 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:29.596049 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:29.585250 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:48:29.623124 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:29.623084 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": context deadline exceeded" Nov 02 01:48:29.623803 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:29.623766 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:48:29.702857 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:29.702745 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 01:48:29.703486 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:29.703461 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:48:29.781694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:29.760470 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:29.782090 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:29.768773 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 02 01:48:29.799697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:29.782607 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:48:29.979812 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:29.971247 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:30.006825 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:30.006779 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 02 01:48:30.135721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:30.020261 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:48:30.135721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:30.020330 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:48:30.135721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:30.021715 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:30.135721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:30.022028 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:30.135721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:30.052490 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:48:30.135721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:30.052560 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:48:30.135721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:30.054294 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 01:48:30.135721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:30.055378 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:48:30.135721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:30.119238 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 02 01:48:30.135721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:30.133378 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:48:30.674695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:30.598764 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 01:48:30.674695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:30.600093 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:30.674695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:30.600752 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 02 01:48:30.674695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:30.601313 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:30.674695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:30.639526 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": context deadline exceeded" Nov 02 01:48:30.753906 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:30.707007 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 01:48:30.794692 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:30.785692 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 02 01:48:31.025788 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:30.990501 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 02 01:48:31.124098 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:31.047296 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:31.150688 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:31.047348 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Nov 02 01:48:31.150688 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:31.082337 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 01:48:31.150688 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:31.123183 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:31.150688 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:31.123247 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 01:48:31.714734 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:31.657856 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:31.715447 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:31.715395 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 01:48:31.886710 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:31.715632 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 01:48:31.911733 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:31.715661 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Nov 02 01:48:32.017655 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:31.715702 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 02 01:48:32.081227 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:31.768625 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:32.082153 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:31.827257 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 02 01:48:32.150571 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:32.150528 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 02 01:48:32.150957 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:32.150929 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:32.151431 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:32.151391 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:32.152426 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:32.152403 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:32.152810 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:32.152774 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Nov 02 01:48:32.153137 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:32.153117 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 01:48:32.153458 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:32.153437 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 02 01:48:32.200092 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:32.188009 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:48:32.200092 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:32.188072 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" podUID="652e50af-8a91-43ff-bb49-bbdf781feefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:32.725580 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:32.710564 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": context deadline exceeded" Nov 02 01:48:32.941707 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:32.933412 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 01:48:32.947531 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:32.946117 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 01:48:32.968695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:32.956744 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:33.211815 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:33.022824 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:33.241789 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:33.082290 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 01:48:33.315695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:33.083897 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 02 01:48:33.322694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:33.105954 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:33.326787 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:33.176202 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:48:33.327043 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:33.327010 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" podUID="98be750b-4e74-4378-871b-3a3e9d53cd78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:33.327195 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:33.223518 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:33.347719 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:33.208526 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:33.348382 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:33.223703 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 02 01:48:33.348862 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:33.223840 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 01:48:33.349304 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:33.224073 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 01:48:33.349791 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:33.224085 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:33.648890 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:33.648842 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded" Nov 02 01:48:33.791151 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:33.752480 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:33.884855 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:33.884806 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:33.949427 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:33.949310 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:48:33.949738 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:33.949699 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" podUID="c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:34.046763 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.037428 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 01:48:34.046763 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.039277 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 01:48:34.200770 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.194077 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:48:34.221689 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.220781 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" podUID="29375ec5-a360-4efd-8a4d-9590d0337a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:34.237956 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.237915 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:34.276692 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.273613 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:48:34.276692 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.273711 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" podUID="54c547c6-30fd-43be-ac27-299da9065a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:34.363898 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.363853 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:34.368780 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.368744 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 01:48:34.371097 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.371060 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:34.373860 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.371319 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 02 01:48:34.374222 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.374197 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 01:48:34.374530 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.371379 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 02 01:48:34.374712 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.371403 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 02 01:48:34.381691 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.381646 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:48:34.493194 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.492989 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Nov 02 01:48:34.530594 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:48:34.530559 5044 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.086s" Nov 02 01:48:34.531284 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.531266 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:48:34.531471 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.531460 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:48:34.533132 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.532460 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:48:34.533319 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.533298 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:48:34.650849 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.650797 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:48:34.770689 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.760373 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:48:34.808360 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.808332 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:48:34.829719 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.829605 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:48:34.872047 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.872018 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:48:34.882393 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.882367 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:48:34.888264 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.888109 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:48:34.904911 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.904887 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:48:34.920968 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:34.919612 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:48:35.057283 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:35.057192 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:35.111196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:35.111164 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm"] Nov 02 01:48:35.111867 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:35.111849 5044 cpu_manager.go:414] "containerMap: removing container" podUID="d0d78e2c-221f-47f2-966f-bd42c211bae9" containerName="revoke-keys" Nov 02 01:48:35.111867 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:35.111866 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d78e2c-221f-47f2-966f-bd42c211bae9" containerName="revoke-keys" Nov 02 01:48:35.112023 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:35.111984 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d78e2c-221f-47f2-966f-bd42c211bae9" containerName="revoke-keys" Nov 02 01:48:35.115477 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:35.115457 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm" Nov 02 01:48:35.181707 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:35.171355 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 01:48:35.181707 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:35.171405 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\"" Nov 02 01:48:35.181707 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:35.174501 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm"] Nov 02 01:48:35.222529 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:35.222483 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhnbq\" (UniqueName: \"kubernetes.io/projected/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-kube-api-access-jhnbq\") pod \"maas-api-revoke-sub-019f3a91-gdgcm\" (UID: \"84dd5a1d-8f13-49f7-9d97-16952e86ee5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm" Nov 02 01:48:35.222755 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:35.222594 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-019f3a91-gdgcm\" (UID: \"84dd5a1d-8f13-49f7-9d97-16952e86ee5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm" Nov 02 01:48:35.222755 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:35.222645 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-019f3a91-gdgcm\" (UID: \"84dd5a1d-8f13-49f7-9d97-16952e86ee5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm" Nov 02 01:48:35.324022 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:35.323943 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhnbq\" (UniqueName: \"kubernetes.io/projected/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-kube-api-access-jhnbq\") pod \"maas-api-revoke-sub-019f3a91-gdgcm\" (UID: \"84dd5a1d-8f13-49f7-9d97-16952e86ee5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm" Nov 02 01:48:35.324184 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:35.324023 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-019f3a91-gdgcm\" (UID: \"84dd5a1d-8f13-49f7-9d97-16952e86ee5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm" Nov 02 01:48:35.324184 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:35.324042 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-019f3a91-gdgcm\" (UID: \"84dd5a1d-8f13-49f7-9d97-16952e86ee5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm" Nov 02 01:48:35.324631 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:35.324609 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-019f3a91-gdgcm\" (UID: \"84dd5a1d-8f13-49f7-9d97-16952e86ee5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm" Nov 02 01:48:35.326351 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:35.326327 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-019f3a91-gdgcm\" (UID: \"84dd5a1d-8f13-49f7-9d97-16952e86ee5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm" Nov 02 01:48:35.387027 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:35.386997 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhnbq\" (UniqueName: \"kubernetes.io/projected/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-kube-api-access-jhnbq\") pod \"maas-api-revoke-sub-019f3a91-gdgcm\" (UID: \"84dd5a1d-8f13-49f7-9d97-16952e86ee5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm" Nov 02 01:48:35.429419 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:35.429389 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm" Nov 02 01:48:35.826296 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:35.826164 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm"] Nov 02 01:48:36.831826 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:36.831791 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm" event={"ID":"84dd5a1d-8f13-49f7-9d97-16952e86ee5f","Type":"ContainerStarted","Data":"5250a703d6cd7d9dbb8a3645fd78be98119f4bca728f658dfa68a3f29d312f2f"} Nov 02 01:48:36.832218 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:36.831849 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm" event={"ID":"84dd5a1d-8f13-49f7-9d97-16952e86ee5f","Type":"ContainerStarted","Data":"52fec8aaf53dc85cbff2adb4423fe5c732bc608d8ca36afb30abb8a1a3007bcf"} Nov 02 01:48:36.951175 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:36.951106 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm" podStartSLOduration=1.951092751 podStartE2EDuration="1.951092751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:36.948844057 +0000 UTC m=+1433.620096738" watchObservedRunningTime="2026-11-02 01:48:36.951092751 +0000 UTC m=+1433.622345436" Nov 02 01:48:37.840985 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:37.840953 5044 generic.go:356] "Generic (PLEG): container finished" podID="84dd5a1d-8f13-49f7-9d97-16952e86ee5f" containerID="5250a703d6cd7d9dbb8a3645fd78be98119f4bca728f658dfa68a3f29d312f2f" exitCode=0 Nov 02 01:48:37.841443 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:37.841033 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm" event={"ID":"84dd5a1d-8f13-49f7-9d97-16952e86ee5f","Type":"ContainerDied","Data":"5250a703d6cd7d9dbb8a3645fd78be98119f4bca728f658dfa68a3f29d312f2f"} Nov 02 01:48:39.030619 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.030593 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm" Nov 02 01:48:39.079264 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.079230 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-kube-api-access-jhnbq\" (UniqueName: \"kubernetes.io/projected/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-kube-api-access-jhnbq\") pod \"84dd5a1d-8f13-49f7-9d97-16952e86ee5f\" (UID: \"84dd5a1d-8f13-49f7-9d97-16952e86ee5f\") " Nov 02 01:48:39.079475 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.079455 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-maas-api-xks-ca\") pod \"84dd5a1d-8f13-49f7-9d97-16952e86ee5f\" (UID: \"84dd5a1d-8f13-49f7-9d97-16952e86ee5f\") " Nov 02 01:48:39.079543 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.079513 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-maas-api-openshift-ca\") pod \"84dd5a1d-8f13-49f7-9d97-16952e86ee5f\" (UID: \"84dd5a1d-8f13-49f7-9d97-16952e86ee5f\") " Nov 02 01:48:39.079966 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.079937 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-maas-api-openshift-ca" pod "84dd5a1d-8f13-49f7-9d97-16952e86ee5f" (UID: "84dd5a1d-8f13-49f7-9d97-16952e86ee5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:48:39.080307 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.080287 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-maas-api-openshift-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:48:39.081448 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.081419 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-kube-api-access-jhnbq" pod "84dd5a1d-8f13-49f7-9d97-16952e86ee5f" (UID: "84dd5a1d-8f13-49f7-9d97-16952e86ee5f"). InnerVolumeSpecName "kube-api-access-jhnbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:48:39.081543 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.081479 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-maas-api-xks-ca" pod "84dd5a1d-8f13-49f7-9d97-16952e86ee5f" (UID: "84dd5a1d-8f13-49f7-9d97-16952e86ee5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:48:39.182039 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.181952 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhnbq\" (UniqueName: \"kubernetes.io/projected/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-kube-api-access-jhnbq\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:48:39.182039 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.181990 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84dd5a1d-8f13-49f7-9d97-16952e86ee5f-maas-api-xks-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:48:39.672714 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.672663 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc"] Nov 02 01:48:39.673321 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.673307 5044 cpu_manager.go:414] "containerMap: removing container" podUID="84dd5a1d-8f13-49f7-9d97-16952e86ee5f" containerName="revoke-keys" Nov 02 01:48:39.673369 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.673322 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84dd5a1d-8f13-49f7-9d97-16952e86ee5f" containerName="revoke-keys" Nov 02 01:48:39.673447 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.673438 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="84dd5a1d-8f13-49f7-9d97-16952e86ee5f" containerName="revoke-keys" Nov 02 01:48:39.677095 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.677078 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.785595 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.785562 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc"] Nov 02 01:48:39.792061 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.792024 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.792220 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.792082 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.792220 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.792183 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.792345 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.792273 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbtfw\" (UniqueName: \"kubernetes.io/projected/44289fed-9af9-4b30-86f2-c4ec83988f44-kube-api-access-sbtfw\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.792345 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.792309 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.792501 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.792443 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.792558 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.792539 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.792614 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.792566 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44289fed-9af9-4b30-86f2-c4ec83988f44-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.792614 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.792599 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.857331 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.857294 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm" Nov 02 01:48:39.857507 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.857330 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm" event={"ID":"84dd5a1d-8f13-49f7-9d97-16952e86ee5f","Type":"ContainerDied","Data":"52fec8aaf53dc85cbff2adb4423fe5c732bc608d8ca36afb30abb8a1a3007bcf"} Nov 02 01:48:39.857507 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.857383 5044 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52fec8aaf53dc85cbff2adb4423fe5c732bc608d8ca36afb30abb8a1a3007bcf" Nov 02 01:48:39.880983 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.880956 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:39.893476 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.893441 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.893476 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.893471 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44289fed-9af9-4b30-86f2-c4ec83988f44-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.893701 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.893495 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.893701 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.893603 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.893701 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.893627 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.893701 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.893654 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.893929 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.893740 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbtfw\" (UniqueName: \"kubernetes.io/projected/44289fed-9af9-4b30-86f2-c4ec83988f44-kube-api-access-sbtfw\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.893929 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.893771 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.893929 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.893886 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-data\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.893929 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.893891 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.894150 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.893955 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-workload-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.894243 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.894206 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-credential-socket\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.894243 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.894227 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-workload-certs\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.894458 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.894436 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44289fed-9af9-4b30-86f2-c4ec83988f44-istiod-ca-cert\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.896091 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.896070 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-envoy\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.896442 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.896417 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-podinfo\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.939762 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.939733 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j"] Nov 02 01:48:39.948516 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.948488 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:39.968371 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.960759 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-token\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.968371 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.966208 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbtfw\" (UniqueName: \"kubernetes.io/projected/44289fed-9af9-4b30-86f2-c4ec83988f44-kube-api-access-sbtfw\") pod \"e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.994996 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.994965 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:39.995172 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.995029 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:39.995289 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.995266 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:39.995430 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.995266 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:39.995502 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.995476 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:39.995579 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.995563 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6523f25-d7c1-4881-8adc-788c8226ea9c-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:39.995641 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.995598 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:39.995641 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.995629 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:39.995829 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.995804 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scm6l\" (UniqueName: \"kubernetes.io/projected/a6523f25-d7c1-4881-8adc-788c8226ea9c-kube-api-access-scm6l\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:39.995912 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:39.995897 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:40.011218 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.011185 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j"] Nov 02 01:48:40.099196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.097861 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6523f25-d7c1-4881-8adc-788c8226ea9c-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:40.099196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.097923 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:40.099196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.097960 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:40.099196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.098067 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scm6l\" (UniqueName: \"kubernetes.io/projected/a6523f25-d7c1-4881-8adc-788c8226ea9c-kube-api-access-scm6l\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:40.099196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.098163 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:40.099196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.098203 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:40.099196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.098272 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:40.099196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.098448 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:40.099196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.098495 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:40.099196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.098778 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6523f25-d7c1-4881-8adc-788c8226ea9c-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:40.105795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.100641 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:40.105795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.100975 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:40.105795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.101257 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:40.105795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.101323 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:40.120855 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.120426 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:40.123758 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.123730 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:40.169751 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.168792 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:40.176894 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.176863 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scm6l\" (UniqueName: \"kubernetes.io/projected/a6523f25-d7c1-4881-8adc-788c8226ea9c-kube-api-access-scm6l\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:40.274092 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.274064 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:40.479094 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:48:40.479060 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44289fed_9af9_4b30_86f2_c4ec83988f44.slice/crio-87dfe53f35864f0d0872a6b6897049dd0026c6163185735950e170dc2ee89eb7 WatchSource:0}: Error finding container 87dfe53f35864f0d0872a6b6897049dd0026c6163185735950e170dc2ee89eb7: Status 404 returned error can't find the container with id 87dfe53f35864f0d0872a6b6897049dd0026c6163185735950e170dc2ee89eb7 Nov 02 01:48:40.479300 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.479279 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc"] Nov 02 01:48:40.481638 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.481609 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:40.481748 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.481690 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:40.481748 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.481724 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:40.658174 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.658142 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j"] Nov 02 01:48:40.658567 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:48:40.658541 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6523f25_d7c1_4881_8adc_788c8226ea9c.slice/crio-2315f423a0826bdc5307134017f63b4d3f14d510c6c70ce30f503af347cc2711 WatchSource:0}: Error finding container 2315f423a0826bdc5307134017f63b4d3f14d510c6c70ce30f503af347cc2711: Status 404 returned error can't find the container with id 2315f423a0826bdc5307134017f63b4d3f14d510c6c70ce30f503af347cc2711 Nov 02 01:48:40.661391 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.661358 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:40.661498 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.661422 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:40.661498 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.661449 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:40.868946 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.868907 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" event={"ID":"44289fed-9af9-4b30-86f2-c4ec83988f44","Type":"ContainerStarted","Data":"393f689d7f563170c7ff9b2fc65c2becabb7f41647ae0ad340935aaf4fce2c4e"} Nov 02 01:48:40.869157 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.868972 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" event={"ID":"44289fed-9af9-4b30-86f2-c4ec83988f44","Type":"ContainerStarted","Data":"87dfe53f35864f0d0872a6b6897049dd0026c6163185735950e170dc2ee89eb7"} Nov 02 01:48:40.870753 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.870725 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" event={"ID":"a6523f25-d7c1-4881-8adc-788c8226ea9c","Type":"ContainerStarted","Data":"43bb1fa53265bd08f4efa69a428409a731aa7bd49866512abd2b119d08c0c711"} Nov 02 01:48:40.870879 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.870774 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" event={"ID":"a6523f25-d7c1-4881-8adc-788c8226ea9c","Type":"ContainerStarted","Data":"2315f423a0826bdc5307134017f63b4d3f14d510c6c70ce30f503af347cc2711"} Nov 02 01:48:40.995810 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.995774 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:40.996704 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:40.996622 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" podStartSLOduration=1.99660774 podStartE2EDuration="1.99660774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:40.990515019 +0000 UTC m=+1437.661767702" watchObservedRunningTime="2026-11-02 01:48:40.99660774 +0000 UTC m=+1437.667860420" Nov 02 01:48:41.040977 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:41.040941 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:41.115505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:41.115454 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" podStartSLOduration=2.115439791 podStartE2EDuration="2.115439791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:41.112801723 +0000 UTC m=+1437.784054417" watchObservedRunningTime="2026-11-02 01:48:41.115439791 +0000 UTC m=+1437.786692472" Nov 02 01:48:41.286690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:41.286127 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:41.304693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:41.303961 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 01:48:41.304693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:41.304037 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" podUID="a6523f25-d7c1-4881-8adc-788c8226ea9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 01:48:42.012099 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:42.012070 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 01:48:42.012354 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:42.012326 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" podUID="44289fed-9af9-4b30-86f2-c4ec83988f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 01:48:42.307747 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:42.293008 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 01:48:42.307747 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:42.293078 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" podUID="a6523f25-d7c1-4881-8adc-788c8226ea9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 01:48:42.999451 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:42.999021 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 01:48:42.999451 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:42.999079 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" podUID="44289fed-9af9-4b30-86f2-c4ec83988f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 01:48:43.320938 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:43.320852 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 01:48:43.321612 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:43.321577 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" podUID="a6523f25-d7c1-4881-8adc-788c8226ea9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 01:48:44.003475 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:44.001067 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 01:48:44.003475 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:44.001144 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" podUID="44289fed-9af9-4b30-86f2-c4ec83988f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 01:48:44.280124 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:44.280089 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 01:48:44.280411 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:44.280380 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" podUID="a6523f25-d7c1-4881-8adc-788c8226ea9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 01:48:44.915914 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:44.915856 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:45.002695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:45.001775 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 01:48:45.002695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:45.001833 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" podUID="44289fed-9af9-4b30-86f2-c4ec83988f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 01:48:45.280024 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:45.277176 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 01:48:45.280024 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:45.277255 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" podUID="a6523f25-d7c1-4881-8adc-788c8226ea9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 01:48:45.998694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:45.997772 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 01:48:45.998694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:45.997849 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" podUID="44289fed-9af9-4b30-86f2-c4ec83988f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 01:48:46.276911 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:46.276872 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 01:48:46.277199 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:46.277170 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" podUID="a6523f25-d7c1-4881-8adc-788c8226ea9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 01:48:46.996231 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:46.996195 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 01:48:46.996418 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:46.996251 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" podUID="44289fed-9af9-4b30-86f2-c4ec83988f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 01:48:47.274720 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:47.274663 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 01:48:47.275196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:47.274746 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" podUID="a6523f25-d7c1-4881-8adc-788c8226ea9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 01:48:47.940210 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:47.940164 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:48.046690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.046452 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:48.047608 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.047582 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:48.048915 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.048892 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:48:48.087034 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.087005 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:48.279339 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.279311 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:48.279839 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.279698 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:48.280450 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.280433 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:48:48.647721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.647564 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp"] Nov 02 01:48:48.653269 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.653245 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:48.696647 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.696617 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp"] Nov 02 01:48:48.706680 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.706638 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-59d70a90-openshift-default-dockercfg-tqcjb\"" Nov 02 01:48:48.720994 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.720966 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:48.832324 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.827412 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zssmn\" (UniqueName: \"kubernetes.io/projected/8cf135c0-f351-450a-99f0-b2120842b6cd-kube-api-access-zssmn\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:48.832829 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.832758 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-podinfo\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:48.833055 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.833026 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-workload-socket\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:48.833198 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.833182 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-credential-socket\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:48.833309 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.833295 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-token\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:48.893735 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.859756 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-envoy\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:48.893735 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.887156 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:48.919951 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.919823 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-workload-certs\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:48.920121 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.919994 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf135c0-f351-450a-99f0-b2120842b6cd-istiod-ca-cert\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:48.920121 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:48.920073 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-data\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:49.021657 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.021615 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zssmn\" (UniqueName: \"kubernetes.io/projected/8cf135c0-f351-450a-99f0-b2120842b6cd-kube-api-access-zssmn\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:49.021903 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.021741 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-podinfo\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:49.021903 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.021785 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-workload-socket\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:49.021903 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.021825 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-credential-socket\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:49.021903 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.021853 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-token\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:49.021903 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.021893 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-envoy\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:49.022253 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.022027 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-workload-certs\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:49.022253 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.022104 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf135c0-f351-450a-99f0-b2120842b6cd-istiod-ca-cert\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:49.022253 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.022146 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-data\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:49.022413 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.022288 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-workload-socket\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:49.022413 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.022344 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-credential-socket\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:49.022569 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.022549 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-data\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:49.022661 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.022629 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-workload-certs\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:49.023046 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.023025 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf135c0-f351-450a-99f0-b2120842b6cd-istiod-ca-cert\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:49.024848 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.024824 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-envoy\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:49.024989 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.024970 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-podinfo\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:49.041354 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.041324 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:49.083812 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.083784 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-token\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:49.085154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.085125 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zssmn\" (UniqueName: \"kubernetes.io/projected/8cf135c0-f351-450a-99f0-b2120842b6cd-kube-api-access-zssmn\") pod \"e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:49.260804 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.260769 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:49.271348 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.270791 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:49.724502 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:48:49.724469 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf135c0_f351_450a_99f0_b2120842b6cd.slice/crio-0f0ffc14a2a938bbd67eb77dae58b5a777b46c3a1d44212e5de0026fee412d45 WatchSource:0}: Error finding container 0f0ffc14a2a938bbd67eb77dae58b5a777b46c3a1d44212e5de0026fee412d45: Status 404 returned error can't find the container with id 0f0ffc14a2a938bbd67eb77dae58b5a777b46c3a1d44212e5de0026fee412d45 Nov 02 01:48:49.724502 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.724493 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp"] Nov 02 01:48:49.727770 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.727737 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:49.727886 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.727816 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:49.727886 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.727861 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:48:49.988585 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.988479 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" event={"ID":"8cf135c0-f351-450a-99f0-b2120842b6cd","Type":"ContainerStarted","Data":"d6e47f3076aebabad242bf6a12361bad610932c2867f4741b90e8e8659766faa"} Nov 02 01:48:49.988585 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:49.988555 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" event={"ID":"8cf135c0-f351-450a-99f0-b2120842b6cd","Type":"ContainerStarted","Data":"0f0ffc14a2a938bbd67eb77dae58b5a777b46c3a1d44212e5de0026fee412d45"} Nov 02 01:48:50.111166 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:50.111113 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" podStartSLOduration=2.111100807 podStartE2EDuration="2.111100807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:48:50.110829983 +0000 UTC m=+1446.782082678" watchObservedRunningTime="2026-11-02 01:48:50.111100807 +0000 UTC m=+1446.782353514" Nov 02 01:48:50.169045 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:50.169006 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:50.271626 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:50.271593 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:50.273460 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:50.273432 5044 patch_prober.go:29] interesting pod/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 01:48:50.273608 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:50.273478 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" podUID="8cf135c0-f351-450a-99f0-b2120842b6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 01:48:50.374374 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:50.374311 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:50.579699 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:50.579603 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:50.969550 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:50.969442 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:51.275706 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:51.275662 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:51.979091 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:51.979049 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:52.004610 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:52.004580 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:52.005801 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:52.005781 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:48:52.291277 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:52.291240 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:52.454160 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:52.454129 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:52.604864 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:52.604748 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:54.175022 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:48:54.174983 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:48:55.297587 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:55.297549 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:55.451291 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:55.450840 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:55.598076 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:55.597991 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:55.742507 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:55.742471 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:55.894891 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:55.894807 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:56.045549 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:56.045518 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:56.739081 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:56.739042 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:56.977426 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:56.977394 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:58.482984 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:58.482952 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:58.698630 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:58.698599 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:48:58.934958 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:48:58.934923 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:00.921221 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:00.921189 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:02.167787 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:02.167757 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:02.360308 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:02.360276 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:02.512702 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:02.512663 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:03.050047 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:03.049990 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:03.224263 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:03.224228 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:05.640689 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:05.638607 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:05.654136 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:05.651811 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:05.654136 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:05.653410 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 01:49:05.751085 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:05.749905 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:05.795564 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:05.794731 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:05.848803 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:05.848759 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:06.089821 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:06.089773 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:06.137809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:06.136768 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:06.160248 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:06.160209 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:06.161023 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:06.160512 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:06.161494 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:06.160541 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 01:49:06.266843 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:06.266802 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:06.546251 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:06.545962 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded" Nov 02 01:49:06.663893 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:06.663853 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": context deadline exceeded" Nov 02 01:49:06.665224 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:06.665203 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:49:06.801533 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:06.799745 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:06.801533 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:06.800500 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:49:07.264757 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:07.237313 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:07.265133 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:07.256766 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:07.265791 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:07.265753 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:49:07.266012 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:07.265998 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:49:07.299206 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:07.257693 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:07.299206 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:07.298241 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:49:07.346811 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:07.323199 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 01:49:07.346811 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:07.323273 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" podUID="f9008e53-2fc7-44ab-b841-e4eea224fded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:07.387718 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:07.384247 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:07.703749 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:07.602107 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:07.711533 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:07.643162 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:07.739963 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:07.643215 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 01:49:07.741208 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:07.644617 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 01:49:07.742216 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:07.687824 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:07.820186 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:07.818151 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:08.173096 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:08.173046 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 02 01:49:08.295697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:08.295119 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 02 01:49:08.330748 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:08.312741 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:08.330748 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:08.315044 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Nov 02 01:49:08.332478 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:08.331042 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 01:49:08.332478 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:08.331211 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 01:49:08.352949 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:08.352897 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:08.386821 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:08.386777 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:08.407700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:08.404423 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:49:08.910596 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:08.910539 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:08.911839 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:08.911041 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 01:49:08.920685 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:08.912343 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:08.920685 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:08.912434 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:49:08.920685 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:08.913067 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:49:08.920685 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:08.918744 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 02 01:49:08.920685 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:08.919617 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:49:08.937693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:08.921089 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 02 01:49:08.941831 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:08.914659 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 01:49:08.941831 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:08.940043 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:49:09.204747 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:09.186053 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:09.245698 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:09.206111 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:49:09.339695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:09.330843 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 02 01:49:09.357695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:09.342275 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 01:49:09.357695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:09.343637 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:49:09.357695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:09.346194 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:09.357695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:09.346525 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:09.377696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:09.371019 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 02 01:49:09.439701 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:09.415147 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:49:09.445807 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:09.445763 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:09.620388 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:09.562313 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:09.999759 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:09.975088 5044 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:10.124653 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:10.010080 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:10.264461 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:10.011992 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 02 01:49:10.264650 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:10.038089 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:10.264773 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:10.085345 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 01:49:10.265097 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:10.184007 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 01:49:10.265097 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:10.265069 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 01:49:10.363845 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:10.363793 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:10.517783 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:10.418791 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:10.517783 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:10.421781 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" podUID="39c708b5-209a-4b5c-9f5b-402174f822b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": context deadline exceeded" Nov 02 01:49:10.517783 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:10.427749 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 01:49:10.517783 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:10.427789 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:10.517783 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:10.427827 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 01:49:10.547795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:10.427863 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 02 01:49:11.016753 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:11.003872 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 01:49:11.016753 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:11.003936 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 01:49:11.016753 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:11.003969 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4" Nov 02 01:49:11.016753 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:11.005022 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 01:49:11.053577 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:11.052696 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 01:49:11.064818 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:11.064794 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 01:49:11.079037 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:11.079010 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 01:49:11.147307 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:11.147265 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 01:49:11.152593 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:11.151778 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 01:49:11.152593 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:11.152429 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 01:49:11.158825 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:11.158800 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 01:49:11.434056 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:11.433960 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:13.997992 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:13.997958 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:14.928849 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:14.928815 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:15.075280 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:15.075234 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:15.218374 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:15.218289 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:16.250356 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.250281 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:16.410615 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.410579 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:16.435336 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.435301 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x"] Nov 02 01:49:16.442422 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.442395 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.526916 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.526880 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x"] Nov 02 01:49:16.550216 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.550187 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 01:49:16.550477 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.550187 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 01:49:16.550617 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.550227 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 01:49:16.550699 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.550280 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6ghtm\"" Nov 02 01:49:16.579828 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.579784 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9655a38-bc2f-498e-b3ea-b2d17e408844-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.580435 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.580411 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mlnk\" (UniqueName: \"kubernetes.io/projected/e9655a38-bc2f-498e-b3ea-b2d17e408844-kube-api-access-6mlnk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.580781 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.580755 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9655a38-bc2f-498e-b3ea-b2d17e408844-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.580925 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.580816 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9655a38-bc2f-498e-b3ea-b2d17e408844-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.580925 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.580850 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9655a38-bc2f-498e-b3ea-b2d17e408844-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.581143 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.581112 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9655a38-bc2f-498e-b3ea-b2d17e408844-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.582192 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.582164 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9655a38-bc2f-498e-b3ea-b2d17e408844-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.683765 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.683721 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9655a38-bc2f-498e-b3ea-b2d17e408844-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.684003 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.683793 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9655a38-bc2f-498e-b3ea-b2d17e408844-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.684003 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.683924 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9655a38-bc2f-498e-b3ea-b2d17e408844-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.684003 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.683991 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mlnk\" (UniqueName: \"kubernetes.io/projected/e9655a38-bc2f-498e-b3ea-b2d17e408844-kube-api-access-6mlnk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.684147 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.684037 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9655a38-bc2f-498e-b3ea-b2d17e408844-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.684147 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.684072 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9655a38-bc2f-498e-b3ea-b2d17e408844-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.684147 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.684105 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9655a38-bc2f-498e-b3ea-b2d17e408844-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.685243 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.685217 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9655a38-bc2f-498e-b3ea-b2d17e408844-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.687380 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.687356 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9655a38-bc2f-498e-b3ea-b2d17e408844-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.687608 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.687591 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9655a38-bc2f-498e-b3ea-b2d17e408844-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.689294 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.689269 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9655a38-bc2f-498e-b3ea-b2d17e408844-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.689449 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.689309 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9655a38-bc2f-498e-b3ea-b2d17e408844-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.705536 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.703509 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:16.749780 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.749752 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9655a38-bc2f-498e-b3ea-b2d17e408844-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.750090 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.750064 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mlnk\" (UniqueName: \"kubernetes.io/projected/e9655a38-bc2f-498e-b3ea-b2d17e408844-kube-api-access-6mlnk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n7g9x\" (UID: \"e9655a38-bc2f-498e-b3ea-b2d17e408844\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:16.765464 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:16.765437 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:17.204014 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:17.203981 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x"] Nov 02 01:49:17.205932 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:17.205915 5044 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:49:17.303258 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:17.303208 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" event={"ID":"e9655a38-bc2f-498e-b3ea-b2d17e408844","Type":"ContainerStarted","Data":"8fff4836ba549457fd9aedea81e46af3641fb60fe66cb5787b87f92b38e25583"} Nov 02 01:49:21.193535 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:21.193499 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:21.382156 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:21.382121 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:49:21.382246 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:21.382189 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:49:22.348037 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:22.348002 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" event={"ID":"e9655a38-bc2f-498e-b3ea-b2d17e408844","Type":"ContainerStarted","Data":"6e571ee25b358830c908dc03afc7a2833ae97327357b215f28a6cd7393ee0264"} Nov 02 01:49:22.348440 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:22.348065 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:22.349867 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:22.349844 5044 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-n7g9x container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:49:22.349976 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:22.349889 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" podUID="e9655a38-bc2f-498e-b3ea-b2d17e408844" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:49:22.564068 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:22.564015 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" podStartSLOduration=2.389170849 podStartE2EDuration="6.56399821s" totalImagesPullingTime="4.174827361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:22.521944299 +0000 UTC m=+1479.193196980" watchObservedRunningTime="2026-11-02 01:49:22.56399821 +0000 UTC m=+1479.235250890" Nov 02 01:49:23.356554 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:23.356524 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n7g9x" Nov 02 01:49:24.176965 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:49:24.176925 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:49:25.377093 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.377055 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm"] Nov 02 01:49:25.383062 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.383040 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.420081 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.420057 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.420197 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.420089 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.420249 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.420186 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2vk7\" (UniqueName: \"kubernetes.io/projected/e8266d54-e810-4f6d-8720-f903ee12ce08-kube-api-access-z2vk7\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.420367 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.420348 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8266d54-e810-4f6d-8720-f903ee12ce08-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.420405 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.420379 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.420554 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.420535 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.420627 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.420582 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.420737 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.420723 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.420801 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.420766 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.438931 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.438901 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm"] Nov 02 01:49:25.522055 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.522024 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.522225 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.522079 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.522225 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.522163 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.522225 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.522205 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.522414 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.522255 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.522414 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.522286 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.522414 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.522334 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2vk7\" (UniqueName: \"kubernetes.io/projected/e8266d54-e810-4f6d-8720-f903ee12ce08-kube-api-access-z2vk7\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.522573 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.522492 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8266d54-e810-4f6d-8720-f903ee12ce08-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.522573 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.522536 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.522735 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.522573 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-data\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.522837 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.522816 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-credential-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.522958 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.522933 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-workload-certs\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.523199 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.523176 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-workload-socket\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.523586 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.523562 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8266d54-e810-4f6d-8720-f903ee12ce08-istiod-ca-cert\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.525379 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.525358 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-podinfo\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.525492 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.525416 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-envoy\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.584493 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.584458 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-token\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.584859 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.584827 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2vk7\" (UniqueName: \"kubernetes.io/projected/e8266d54-e810-4f6d-8720-f903ee12ce08-kube-api-access-z2vk7\") pod \"e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:25.700529 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:25.700442 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:26.015344 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:26.015314 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:26.144870 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:49:26.144833 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8266d54_e810_4f6d_8720_f903ee12ce08.slice/crio-ad22594bd2c4b3a8e087c42dad4bc81eb1b2f19ea433924b9f13f52362517987 WatchSource:0}: Error finding container ad22594bd2c4b3a8e087c42dad4bc81eb1b2f19ea433924b9f13f52362517987: Status 404 returned error can't find the container with id ad22594bd2c4b3a8e087c42dad4bc81eb1b2f19ea433924b9f13f52362517987 Nov 02 01:49:26.145472 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:26.145383 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm"] Nov 02 01:49:26.148039 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:26.147992 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:49:26.148176 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:26.148086 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:49:26.148176 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:26.148119 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:49:26.386568 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:26.386467 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" event={"ID":"e8266d54-e810-4f6d-8720-f903ee12ce08","Type":"ContainerStarted","Data":"fccaaeb93f7e4600725c391382c0d4b133386548512a049172f50f83f0ec66e5"} Nov 02 01:49:26.386568 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:26.386537 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" event={"ID":"e8266d54-e810-4f6d-8720-f903ee12ce08","Type":"ContainerStarted","Data":"ad22594bd2c4b3a8e087c42dad4bc81eb1b2f19ea433924b9f13f52362517987"} Nov 02 01:49:26.510931 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:26.510879 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" podStartSLOduration=1.510865025 podStartE2EDuration="1.510865025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:26.508803644 +0000 UTC m=+1483.180056330" watchObservedRunningTime="2026-11-02 01:49:26.510865025 +0000 UTC m=+1483.182117706" Nov 02 01:49:26.702418 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:26.701188 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:27.702458 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:27.702423 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 01:49:27.702993 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:27.702485 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" podUID="e8266d54-e810-4f6d-8720-f903ee12ce08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 01:49:28.701066 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:28.701029 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 01:49:28.701248 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:28.701098 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" podUID="e8266d54-e810-4f6d-8720-f903ee12ce08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 01:49:29.513167 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:29.513137 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:29.701415 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:29.701368 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 01:49:29.701569 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:29.701444 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" podUID="e8266d54-e810-4f6d-8720-f903ee12ce08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 01:49:30.700835 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:30.700795 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 01:49:30.701218 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:30.700865 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" podUID="e8266d54-e810-4f6d-8720-f903ee12ce08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 01:49:31.018798 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:31.018762 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:31.700829 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:31.700789 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 01:49:31.701106 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:31.700859 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" podUID="e8266d54-e810-4f6d-8720-f903ee12ce08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 01:49:32.700931 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:32.700890 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 01:49:32.701149 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:32.700957 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" podUID="e8266d54-e810-4f6d-8720-f903ee12ce08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 01:49:33.705316 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:33.705279 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:33.705748 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:33.705606 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:33.706506 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:33.706486 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:49:35.526809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:35.526772 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:35.989790 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:35.989751 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:40.969207 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:40.969171 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w"] Nov 02 01:49:40.973508 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:40.973484 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" Nov 02 01:49:41.012532 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:41.012502 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93a422c9-9b80-4152-a286-ff6710ec21e7-maas-api-tls\") pod \"maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w\" (UID: \"93a422c9-9b80-4152-a286-ff6710ec21e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" Nov 02 01:49:41.012690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:41.012553 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93a422c9-9b80-4152-a286-ff6710ec21e7-metrics-certs\") pod \"maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w\" (UID: \"93a422c9-9b80-4152-a286-ff6710ec21e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" Nov 02 01:49:41.012781 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:41.012738 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8qkg\" (UniqueName: \"kubernetes.io/projected/93a422c9-9b80-4152-a286-ff6710ec21e7-kube-api-access-l8qkg\") pod \"maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w\" (UID: \"93a422c9-9b80-4152-a286-ff6710ec21e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" Nov 02 01:49:41.012877 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:41.012860 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a422c9-9b80-4152-a286-ff6710ec21e7-openshift-service-ca\") pod \"maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w\" (UID: \"93a422c9-9b80-4152-a286-ff6710ec21e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" Nov 02 01:49:41.032617 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:41.032589 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w"] Nov 02 01:49:41.077350 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:41.077324 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-59d70a90-serving-cert\"" Nov 02 01:49:41.122609 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:41.122568 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8qkg\" (UniqueName: \"kubernetes.io/projected/93a422c9-9b80-4152-a286-ff6710ec21e7-kube-api-access-l8qkg\") pod \"maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w\" (UID: \"93a422c9-9b80-4152-a286-ff6710ec21e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" Nov 02 01:49:41.122851 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:41.122710 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a422c9-9b80-4152-a286-ff6710ec21e7-openshift-service-ca\") pod \"maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w\" (UID: \"93a422c9-9b80-4152-a286-ff6710ec21e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" Nov 02 01:49:41.122915 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:41.122897 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93a422c9-9b80-4152-a286-ff6710ec21e7-maas-api-tls\") pod \"maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w\" (UID: \"93a422c9-9b80-4152-a286-ff6710ec21e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" Nov 02 01:49:41.122973 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:41.122962 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93a422c9-9b80-4152-a286-ff6710ec21e7-metrics-certs\") pod \"maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w\" (UID: \"93a422c9-9b80-4152-a286-ff6710ec21e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" Nov 02 01:49:41.135932 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:49:41.135902 5044 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-serving-cert: secret "maas-api-e2e-ait-59d70a90-serving-cert" not found Nov 02 01:49:41.136086 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:49:41.136003 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a422c9-9b80-4152-a286-ff6710ec21e7-maas-api-tls podName:93a422c9-9b80-4152-a286-ff6710ec21e7 nodeName:}" failed. No retries permitted until 2026-11-02 01:49:41.635979984 +0000 UTC m=+1498.307232645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93a422c9-9b80-4152-a286-ff6710ec21e7-maas-api-tls") pod "maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" (UID: "93a422c9-9b80-4152-a286-ff6710ec21e7") : secret "maas-api-e2e-ait-59d70a90-serving-cert" not found Nov 02 01:49:41.136393 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:41.136370 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a422c9-9b80-4152-a286-ff6710ec21e7-openshift-service-ca\") pod \"maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w\" (UID: \"93a422c9-9b80-4152-a286-ff6710ec21e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" Nov 02 01:49:41.144024 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:41.144001 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93a422c9-9b80-4152-a286-ff6710ec21e7-metrics-certs\") pod \"maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w\" (UID: \"93a422c9-9b80-4152-a286-ff6710ec21e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" Nov 02 01:49:41.169433 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:41.169405 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:41.200103 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:41.200076 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8qkg\" (UniqueName: \"kubernetes.io/projected/93a422c9-9b80-4152-a286-ff6710ec21e7-kube-api-access-l8qkg\") pod \"maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w\" (UID: \"93a422c9-9b80-4152-a286-ff6710ec21e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" Nov 02 01:49:41.732307 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:41.732274 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93a422c9-9b80-4152-a286-ff6710ec21e7-maas-api-tls\") pod \"maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w\" (UID: \"93a422c9-9b80-4152-a286-ff6710ec21e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" Nov 02 01:49:41.734700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:41.734657 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93a422c9-9b80-4152-a286-ff6710ec21e7-maas-api-tls\") pod \"maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w\" (UID: \"93a422c9-9b80-4152-a286-ff6710ec21e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" Nov 02 01:49:41.891459 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:41.891429 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" Nov 02 01:49:42.274124 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:49:42.274087 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a422c9_9b80_4152_a286_ff6710ec21e7.slice/crio-335cbd38e3c4d1c5f48783f8d400a66aa3f6c9dfb0876829c674cc32840d46fb WatchSource:0}: Error finding container 335cbd38e3c4d1c5f48783f8d400a66aa3f6c9dfb0876829c674cc32840d46fb: Status 404 returned error can't find the container with id 335cbd38e3c4d1c5f48783f8d400a66aa3f6c9dfb0876829c674cc32840d46fb Nov 02 01:49:42.274501 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:42.274481 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w"] Nov 02 01:49:42.540224 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:42.540133 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" event={"ID":"93a422c9-9b80-4152-a286-ff6710ec21e7","Type":"ContainerStarted","Data":"335cbd38e3c4d1c5f48783f8d400a66aa3f6c9dfb0876829c674cc32840d46fb"} Nov 02 01:49:42.817439 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:42.817353 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:43.549051 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:43.549018 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" event={"ID":"93a422c9-9b80-4152-a286-ff6710ec21e7","Type":"ContainerStarted","Data":"4a5ee637ce736b3b554a77f7c72e192a0743b89a63d9a4f07d4a89941ef4d678"} Nov 02 01:49:43.549511 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:43.549254 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" Nov 02 01:49:43.621604 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:43.621529 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 01:49:43.627050 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:43.627027 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 01:49:43.672795 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:43.672733 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" podStartSLOduration=2.639551041 podStartE2EDuration="3.672720167s" totalImagesPullingTime="1.033169126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:43.669964167 +0000 UTC m=+1500.341216845" watchObservedRunningTime="2026-11-02 01:49:43.672720167 +0000 UTC m=+1500.343972848" Nov 02 01:49:43.811505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:43.811472 5044 scope.go:122] "RemoveContainer" containerID="a928a201d99ab8682eabfb09e1d52bf1ba1256a6ada23d98ff559b31a6d926f8" Nov 02 01:49:46.106715 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:46.106661 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:46.267847 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:46.267817 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:46.435165 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:46.435064 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:46.828136 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:46.828107 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:47.017811 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:47.017773 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:49.569968 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:49.569936 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" Nov 02 01:49:51.033443 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.033410 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm"] Nov 02 01:49:51.037485 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.037466 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" Nov 02 01:49:51.091092 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.091054 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-fd723454-serving-cert\"" Nov 02 01:49:51.146481 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.146447 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm"] Nov 02 01:49:51.187400 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.187365 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40c0608c-9731-45bf-996a-c21f0fe4e54b-metrics-certs\") pod \"maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm\" (UID: \"40c0608c-9731-45bf-996a-c21f0fe4e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" Nov 02 01:49:51.187892 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.187864 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40c0608c-9731-45bf-996a-c21f0fe4e54b-maas-api-tls\") pod \"maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm\" (UID: \"40c0608c-9731-45bf-996a-c21f0fe4e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" Nov 02 01:49:51.188154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.188133 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40c0608c-9731-45bf-996a-c21f0fe4e54b-openshift-service-ca\") pod \"maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm\" (UID: \"40c0608c-9731-45bf-996a-c21f0fe4e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" Nov 02 01:49:51.189010 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.188991 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbp6w\" (UniqueName: \"kubernetes.io/projected/40c0608c-9731-45bf-996a-c21f0fe4e54b-kube-api-access-mbp6w\") pod \"maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm\" (UID: \"40c0608c-9731-45bf-996a-c21f0fe4e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" Nov 02 01:49:51.230119 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.230079 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:51.292613 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.292524 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40c0608c-9731-45bf-996a-c21f0fe4e54b-metrics-certs\") pod \"maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm\" (UID: \"40c0608c-9731-45bf-996a-c21f0fe4e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" Nov 02 01:49:51.293209 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.293179 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40c0608c-9731-45bf-996a-c21f0fe4e54b-maas-api-tls\") pod \"maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm\" (UID: \"40c0608c-9731-45bf-996a-c21f0fe4e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" Nov 02 01:49:51.293358 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.293331 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40c0608c-9731-45bf-996a-c21f0fe4e54b-openshift-service-ca\") pod \"maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm\" (UID: \"40c0608c-9731-45bf-996a-c21f0fe4e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" Nov 02 01:49:51.293438 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.293368 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbp6w\" (UniqueName: \"kubernetes.io/projected/40c0608c-9731-45bf-996a-c21f0fe4e54b-kube-api-access-mbp6w\") pod \"maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm\" (UID: \"40c0608c-9731-45bf-996a-c21f0fe4e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" Nov 02 01:49:51.293508 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:49:51.293457 5044 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-serving-cert: secret "maas-api-e2e-derive-fd723454-serving-cert" not found Nov 02 01:49:51.293565 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:49:51.293543 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40c0608c-9731-45bf-996a-c21f0fe4e54b-maas-api-tls podName:40c0608c-9731-45bf-996a-c21f0fe4e54b nodeName:}" failed. No retries permitted until 2026-11-02 01:49:51.793519271 +0000 UTC m=+1508.464771947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40c0608c-9731-45bf-996a-c21f0fe4e54b-maas-api-tls") pod "maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" (UID: "40c0608c-9731-45bf-996a-c21f0fe4e54b") : secret "maas-api-e2e-derive-fd723454-serving-cert" not found Nov 02 01:49:51.294433 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.294408 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40c0608c-9731-45bf-996a-c21f0fe4e54b-openshift-service-ca\") pod \"maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm\" (UID: \"40c0608c-9731-45bf-996a-c21f0fe4e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" Nov 02 01:49:51.299782 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.299647 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40c0608c-9731-45bf-996a-c21f0fe4e54b-metrics-certs\") pod \"maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm\" (UID: \"40c0608c-9731-45bf-996a-c21f0fe4e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" Nov 02 01:49:51.355793 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.355759 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbp6w\" (UniqueName: \"kubernetes.io/projected/40c0608c-9731-45bf-996a-c21f0fe4e54b-kube-api-access-mbp6w\") pod \"maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm\" (UID: \"40c0608c-9731-45bf-996a-c21f0fe4e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" Nov 02 01:49:51.390539 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.390509 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:51.536969 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.536925 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:51.803549 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.803517 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40c0608c-9731-45bf-996a-c21f0fe4e54b-maas-api-tls\") pod \"maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm\" (UID: \"40c0608c-9731-45bf-996a-c21f0fe4e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" Nov 02 01:49:51.805976 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.805953 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40c0608c-9731-45bf-996a-c21f0fe4e54b-maas-api-tls\") pod \"maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm\" (UID: \"40c0608c-9731-45bf-996a-c21f0fe4e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" Nov 02 01:49:51.923988 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.923956 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:51.950730 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:51.950696 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" Nov 02 01:49:52.172780 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:52.161898 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:52.370778 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:52.360166 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm"] Nov 02 01:49:52.377767 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:49:52.377326 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c0608c_9731_45bf_996a_c21f0fe4e54b.slice/crio-69ea016a901b3d9e7544f7631ac0980e659de28ef3a4fe10cfc1492d3cc9c89a WatchSource:0}: Error finding container 69ea016a901b3d9e7544f7631ac0980e659de28ef3a4fe10cfc1492d3cc9c89a: Status 404 returned error can't find the container with id 69ea016a901b3d9e7544f7631ac0980e659de28ef3a4fe10cfc1492d3cc9c89a Nov 02 01:49:52.623626 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:52.623587 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" event={"ID":"40c0608c-9731-45bf-996a-c21f0fe4e54b","Type":"ContainerStarted","Data":"69ea016a901b3d9e7544f7631ac0980e659de28ef3a4fe10cfc1492d3cc9c89a"} Nov 02 01:49:54.176603 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:49:54.176548 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:49:54.649190 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:54.649156 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" event={"ID":"40c0608c-9731-45bf-996a-c21f0fe4e54b","Type":"ContainerStarted","Data":"cf2b2b2ddc089a972eb6d0d1063edb15aba97c86926180dcb34ef6c03e49e346"} Nov 02 01:49:54.649448 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:54.649256 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" Nov 02 01:49:54.771646 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:54.771591 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" podStartSLOduration=3.024664131 podStartE2EDuration="4.77157305s" totalImagesPullingTime="1.746908919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:49:54.76988788 +0000 UTC m=+1511.441140563" watchObservedRunningTime="2026-11-02 01:49:54.77157305 +0000 UTC m=+1511.442825764" Nov 02 01:49:56.477196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:56.477160 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:57.054050 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:57.054018 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:49:57.330836 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:49:57.330742 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:00.663058 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:00.663021 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" Nov 02 01:50:00.776745 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:00.776707 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:00.927853 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:00.927761 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:01.234910 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.234832 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9"] Nov 02 01:50:01.235318 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.235291 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:01.239969 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.239947 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" Nov 02 01:50:01.296510 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.296479 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5fc56410-serving-cert\"" Nov 02 01:50:01.301925 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.301903 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9"] Nov 02 01:50:01.332344 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.332301 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a827d2-d8ce-4cc8-b440-c171c094c1da-maas-api-tls\") pod \"maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9\" (UID: \"01a827d2-d8ce-4cc8-b440-c171c094c1da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" Nov 02 01:50:01.332495 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.332380 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a827d2-d8ce-4cc8-b440-c171c094c1da-openshift-service-ca\") pod \"maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9\" (UID: \"01a827d2-d8ce-4cc8-b440-c171c094c1da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" Nov 02 01:50:01.332495 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.332410 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s75p\" (UniqueName: \"kubernetes.io/projected/01a827d2-d8ce-4cc8-b440-c171c094c1da-kube-api-access-2s75p\") pod \"maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9\" (UID: \"01a827d2-d8ce-4cc8-b440-c171c094c1da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" Nov 02 01:50:01.332495 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.332445 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01a827d2-d8ce-4cc8-b440-c171c094c1da-metrics-certs\") pod \"maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9\" (UID: \"01a827d2-d8ce-4cc8-b440-c171c094c1da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" Nov 02 01:50:01.432643 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.432613 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:01.433475 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.433455 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a827d2-d8ce-4cc8-b440-c171c094c1da-openshift-service-ca\") pod \"maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9\" (UID: \"01a827d2-d8ce-4cc8-b440-c171c094c1da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" Nov 02 01:50:01.433545 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.433483 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s75p\" (UniqueName: \"kubernetes.io/projected/01a827d2-d8ce-4cc8-b440-c171c094c1da-kube-api-access-2s75p\") pod \"maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9\" (UID: \"01a827d2-d8ce-4cc8-b440-c171c094c1da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" Nov 02 01:50:01.433545 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.433509 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01a827d2-d8ce-4cc8-b440-c171c094c1da-metrics-certs\") pod \"maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9\" (UID: \"01a827d2-d8ce-4cc8-b440-c171c094c1da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" Nov 02 01:50:01.433801 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.433785 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a827d2-d8ce-4cc8-b440-c171c094c1da-maas-api-tls\") pod \"maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9\" (UID: \"01a827d2-d8ce-4cc8-b440-c171c094c1da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" Nov 02 01:50:01.433953 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:50:01.433931 5044 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-serving-cert: secret "maas-api-e2e-ait-5fc56410-serving-cert" not found Nov 02 01:50:01.434035 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:50:01.434020 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01a827d2-d8ce-4cc8-b440-c171c094c1da-maas-api-tls podName:01a827d2-d8ce-4cc8-b440-c171c094c1da nodeName:}" failed. No retries permitted until 2026-11-02 01:50:01.934000259 +0000 UTC m=+1518.605252931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01a827d2-d8ce-4cc8-b440-c171c094c1da-maas-api-tls") pod "maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" (UID: "01a827d2-d8ce-4cc8-b440-c171c094c1da") : secret "maas-api-e2e-ait-5fc56410-serving-cert" not found Nov 02 01:50:01.434229 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.434210 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a827d2-d8ce-4cc8-b440-c171c094c1da-openshift-service-ca\") pod \"maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9\" (UID: \"01a827d2-d8ce-4cc8-b440-c171c094c1da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" Nov 02 01:50:01.436186 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.436170 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01a827d2-d8ce-4cc8-b440-c171c094c1da-metrics-certs\") pod \"maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9\" (UID: \"01a827d2-d8ce-4cc8-b440-c171c094c1da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" Nov 02 01:50:01.524635 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.524599 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s75p\" (UniqueName: \"kubernetes.io/projected/01a827d2-d8ce-4cc8-b440-c171c094c1da-kube-api-access-2s75p\") pod \"maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9\" (UID: \"01a827d2-d8ce-4cc8-b440-c171c094c1da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" Nov 02 01:50:01.591694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.591305 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:01.750635 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.750587 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:01.942730 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.942598 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a827d2-d8ce-4cc8-b440-c171c094c1da-maas-api-tls\") pod \"maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9\" (UID: \"01a827d2-d8ce-4cc8-b440-c171c094c1da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" Nov 02 01:50:01.945188 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:01.945163 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a827d2-d8ce-4cc8-b440-c171c094c1da-maas-api-tls\") pod \"maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9\" (UID: \"01a827d2-d8ce-4cc8-b440-c171c094c1da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" Nov 02 01:50:02.153845 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:02.153811 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" Nov 02 01:50:02.548030 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:02.548000 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9"] Nov 02 01:50:02.548479 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:50:02.548449 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a827d2_d8ce_4cc8_b440_c171c094c1da.slice/crio-feb573327673202a3650bb000748d8f4863881f14df570e1659aecf507834dca WatchSource:0}: Error finding container feb573327673202a3650bb000748d8f4863881f14df570e1659aecf507834dca: Status 404 returned error can't find the container with id feb573327673202a3650bb000748d8f4863881f14df570e1659aecf507834dca Nov 02 01:50:02.729465 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:02.729427 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" event={"ID":"01a827d2-d8ce-4cc8-b440-c171c094c1da","Type":"ContainerStarted","Data":"feb573327673202a3650bb000748d8f4863881f14df570e1659aecf507834dca"} Nov 02 01:50:03.737790 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:03.737695 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" event={"ID":"01a827d2-d8ce-4cc8-b440-c171c094c1da","Type":"ContainerStarted","Data":"0fcc3b5233e96270385ff059a6029a4dd27818f46462ed1622701b2bae7612d9"} Nov 02 01:50:03.737790 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:03.737771 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" Nov 02 01:50:03.861526 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:03.861471 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" podStartSLOduration=2.015194132 podStartE2EDuration="2.861458069s" totalImagesPullingTime="846.263937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:50:03.857778686 +0000 UTC m=+1520.529031369" watchObservedRunningTime="2026-11-02 01:50:03.861458069 +0000 UTC m=+1520.532710749" Nov 02 01:50:06.587428 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:06.587393 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:07.275472 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:07.275434 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:07.429060 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:07.429026 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:07.574987 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:07.574913 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:07.882931 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:07.882826 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:09.753046 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:09.753017 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" Nov 02 01:50:12.198701 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:12.197428 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:14.193398 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:14.193345 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:14.358156 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:14.358124 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:15.258573 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.258543 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk"] Nov 02 01:50:15.280980 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.280950 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.378023 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.377989 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pjf7g\"" Nov 02 01:50:15.409341 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.409306 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk"] Nov 02 01:50:15.434184 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.434105 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.434547 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.434521 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.434895 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.434872 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.435091 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.435072 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.435264 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.435246 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.435432 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.435414 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scb6h\" (UniqueName: \"kubernetes.io/projected/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-kube-api-access-scb6h\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.435718 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.435663 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.435934 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.435916 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.436134 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.436117 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.537202 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.537121 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.537202 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.537171 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.537445 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.537229 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.537445 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.537253 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.537445 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.537390 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.537445 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.537423 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.537653 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.537446 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.537653 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.537479 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scb6h\" (UniqueName: \"kubernetes.io/projected/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-kube-api-access-scb6h\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.537653 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.537590 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.537927 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.537858 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.537927 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.537871 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.538134 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.537951 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.538134 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.537969 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.538263 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.538220 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.539636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.539619 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.540248 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.540221 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.600367 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.600334 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.602216 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.602186 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scb6h\" (UniqueName: \"kubernetes.io/projected/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-kube-api-access-scb6h\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-pjltk\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:15.606140 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:15.606121 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:16.071577 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:16.071544 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk"] Nov 02 01:50:16.071799 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:50:16.071770 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda89ef7a_b9e2_4ecf_bcf5_dba89ba3dd15.slice/crio-406f61a772a2debc2b2fae0a1808546ad509ff273d7570e4de055f2a05439298 WatchSource:0}: Error finding container 406f61a772a2debc2b2fae0a1808546ad509ff273d7570e4de055f2a05439298: Status 404 returned error can't find the container with id 406f61a772a2debc2b2fae0a1808546ad509ff273d7570e4de055f2a05439298 Nov 02 01:50:16.075079 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:16.075033 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:50:16.075205 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:16.075129 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:50:16.075205 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:16.075174 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:50:16.942244 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:16.942206 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" event={"ID":"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15","Type":"ContainerStarted","Data":"5d246185e91e96a8a1e5dddd099a6b961167a701338521d5548af550570d7ae5"} Nov 02 01:50:16.942608 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:16.942277 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" event={"ID":"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15","Type":"ContainerStarted","Data":"406f61a772a2debc2b2fae0a1808546ad509ff273d7570e4de055f2a05439298"} Nov 02 01:50:17.122555 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:17.122494 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" podStartSLOduration=2.122472874 podStartE2EDuration="2.122472874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:50:17.116557146 +0000 UTC m=+1533.787809827" watchObservedRunningTime="2026-11-02 01:50:17.122472874 +0000 UTC m=+1533.793725556" Nov 02 01:50:17.415548 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:17.415509 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:17.618558 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:17.618519 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:17.621824 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:17.621793 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pjltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 01:50:17.622130 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:17.622100 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" podUID="da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 01:50:18.607134 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:18.607099 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pjltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 01:50:18.607520 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:18.607178 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" podUID="da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 01:50:19.607482 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:19.607446 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pjltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 01:50:19.607890 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:19.607509 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" podUID="da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 01:50:20.606882 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:20.606846 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pjltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 01:50:20.607144 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:20.606912 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" podUID="da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 01:50:21.001435 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:21.001340 5044 generic.go:356] "Generic (PLEG): container finished" podID="8cf135c0-f351-450a-99f0-b2120842b6cd" containerID="d6e47f3076aebabad242bf6a12361bad610932c2867f4741b90e8e8659766faa" exitCode=0 Nov 02 01:50:21.001902 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:21.001416 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" event={"ID":"8cf135c0-f351-450a-99f0-b2120842b6cd","Type":"ContainerDied","Data":"d6e47f3076aebabad242bf6a12361bad610932c2867f4741b90e8e8659766faa"} Nov 02 01:50:21.002080 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:21.002052 5044 scope.go:122] "RemoveContainer" containerID="d6e47f3076aebabad242bf6a12361bad610932c2867f4741b90e8e8659766faa" Nov 02 01:50:21.004020 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:21.003986 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:50:21.004131 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:21.004091 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:50:21.004198 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:21.004144 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:50:21.607175 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:21.607138 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pjltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 01:50:21.607371 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:21.607197 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" podUID="da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 01:50:22.004966 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:22.004880 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:50:22.040337 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:22.040300 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" event={"ID":"8cf135c0-f351-450a-99f0-b2120842b6cd","Type":"ContainerStarted","Data":"1c64f391038f1e370e76548554f1a50f92dac017a87fe13aeeaa7f122b14ff89"} Nov 02 01:50:22.275783 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:22.275748 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:50:22.279880 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:22.279834 5044 patch_prober.go:29] interesting pod/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 01:50:22.280145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:22.280108 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" podUID="8cf135c0-f351-450a-99f0-b2120842b6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 01:50:22.357708 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:22.357647 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:22.613783 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:22.613649 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pjltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 01:50:22.614062 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:22.614030 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" podUID="da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 01:50:23.279104 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:23.279064 5044 patch_prober.go:29] interesting pod/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 01:50:23.279799 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:23.279756 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" podUID="8cf135c0-f351-450a-99f0-b2120842b6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 01:50:23.607720 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:23.607601 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pjltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 01:50:23.607720 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:23.607662 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" podUID="da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 01:50:24.210148 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:50:24.210074 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:50:24.281695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:24.276705 5044 patch_prober.go:29] interesting pod/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 01:50:24.281695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:24.276765 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" podUID="8cf135c0-f351-450a-99f0-b2120842b6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 01:50:24.606992 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:24.606954 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pjltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 02 01:50:24.607203 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:24.607006 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" podUID="da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 02 01:50:25.276024 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:25.275985 5044 patch_prober.go:29] interesting pod/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 01:50:25.276308 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:25.276278 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" podUID="8cf135c0-f351-450a-99f0-b2120842b6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 01:50:25.952477 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:25.952444 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:25.954541 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:25.954514 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:25.955743 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:25.955721 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 01:50:26.273169 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:26.273126 5044 patch_prober.go:29] interesting pod/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 01:50:26.273466 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:26.273435 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" podUID="8cf135c0-f351-450a-99f0-b2120842b6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 01:50:27.176700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:27.176100 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:27.276707 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:27.273529 5044 patch_prober.go:29] interesting pod/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 01:50:27.276707 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:27.273589 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" podUID="8cf135c0-f351-450a-99f0-b2120842b6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 01:50:28.276103 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:28.276076 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:50:28.276543 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:28.276529 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:50:28.277315 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:28.277292 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:50:31.987200 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:31.987151 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:36.809427 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:36.809381 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:41.605091 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:41.605055 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:41.754000 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:41.753950 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:46.677530 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:46.677496 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:52.420400 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:52.420367 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:54.175895 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:50:54.175855 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:50:58.012833 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:58.012779 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:58.374984 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:58.374878 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:50:58.521230 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:50:58.521199 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:02.762679 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:02.762634 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:02.912224 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:02.912187 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:03.229479 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:03.229401 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:09.018375 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:09.018338 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:10.742969 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:10.742932 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:10.891347 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:10.891279 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:11.041706 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:11.041652 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:13.184045 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:13.184001 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:13.331743 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:13.331710 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:14.396135 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:14.392550 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:16.897120 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:16.897069 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:19.673154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:19.673114 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:20.231575 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:20.231531 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:20.377902 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:20.377866 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:24.180875 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:51:24.180832 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:51:24.590849 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:24.590816 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:26.052917 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:26.052882 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:26.199876 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:26.199842 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:29.728142 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:29.728106 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:34.568148 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:34.568102 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:39.319122 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:39.319091 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:42.067835 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:42.067799 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:42.214235 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:42.214201 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:43.386976 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.386931 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w"] Nov 02 01:51:43.387454 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.387162 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" podUID="93a422c9-9b80-4152-a286-ff6710ec21e7" containerName="maas-api" containerID="cri-o://4a5ee637ce736b3b554a77f7c72e192a0743b89a63d9a4f07d4a89941ef4d678" gracePeriod=30 Nov 02 01:51:43.485990 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.485959 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:43.752090 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.752062 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" Nov 02 01:51:43.778316 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.778232 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93a422c9-9b80-4152-a286-ff6710ec21e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93a422c9-9b80-4152-a286-ff6710ec21e7-maas-api-tls\") pod \"93a422c9-9b80-4152-a286-ff6710ec21e7\" (UID: \"93a422c9-9b80-4152-a286-ff6710ec21e7\") " Nov 02 01:51:43.778828 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.778805 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93a422c9-9b80-4152-a286-ff6710ec21e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93a422c9-9b80-4152-a286-ff6710ec21e7-metrics-certs\") pod \"93a422c9-9b80-4152-a286-ff6710ec21e7\" (UID: \"93a422c9-9b80-4152-a286-ff6710ec21e7\") " Nov 02 01:51:43.779060 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.779043 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a422c9-9b80-4152-a286-ff6710ec21e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a422c9-9b80-4152-a286-ff6710ec21e7-openshift-service-ca\") pod \"93a422c9-9b80-4152-a286-ff6710ec21e7\" (UID: \"93a422c9-9b80-4152-a286-ff6710ec21e7\") " Nov 02 01:51:43.779225 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.779193 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a422c9-9b80-4152-a286-ff6710ec21e7-kube-api-access-l8qkg\" (UniqueName: \"kubernetes.io/projected/93a422c9-9b80-4152-a286-ff6710ec21e7-kube-api-access-l8qkg\") pod \"93a422c9-9b80-4152-a286-ff6710ec21e7\" (UID: \"93a422c9-9b80-4152-a286-ff6710ec21e7\") " Nov 02 01:51:43.781452 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.781425 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a422c9-9b80-4152-a286-ff6710ec21e7-openshift-service-ca" pod "93a422c9-9b80-4152-a286-ff6710ec21e7" (UID: "93a422c9-9b80-4152-a286-ff6710ec21e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:51:43.811037 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.809747 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a422c9-9b80-4152-a286-ff6710ec21e7-maas-api-tls" pod "93a422c9-9b80-4152-a286-ff6710ec21e7" (UID: "93a422c9-9b80-4152-a286-ff6710ec21e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:51:43.841118 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.830693 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a422c9-9b80-4152-a286-ff6710ec21e7-metrics-certs" pod "93a422c9-9b80-4152-a286-ff6710ec21e7" (UID: "93a422c9-9b80-4152-a286-ff6710ec21e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:51:43.849916 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.849864 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a422c9-9b80-4152-a286-ff6710ec21e7-kube-api-access-l8qkg" pod "93a422c9-9b80-4152-a286-ff6710ec21e7" (UID: "93a422c9-9b80-4152-a286-ff6710ec21e7"). InnerVolumeSpecName "kube-api-access-l8qkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:51:43.895773 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.895412 5044 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a422c9-9b80-4152-a286-ff6710ec21e7-openshift-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:43.895773 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.895462 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8qkg\" (UniqueName: \"kubernetes.io/projected/93a422c9-9b80-4152-a286-ff6710ec21e7-kube-api-access-l8qkg\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:43.895773 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.895483 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93a422c9-9b80-4152-a286-ff6710ec21e7-maas-api-tls\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:43.895773 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.895498 5044 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93a422c9-9b80-4152-a286-ff6710ec21e7-metrics-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:43.929576 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.926913 5044 generic.go:356] "Generic (PLEG): container finished" podID="93a422c9-9b80-4152-a286-ff6710ec21e7" containerID="4a5ee637ce736b3b554a77f7c72e192a0743b89a63d9a4f07d4a89941ef4d678" exitCode=0 Nov 02 01:51:43.929576 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.927156 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" event={"ID":"93a422c9-9b80-4152-a286-ff6710ec21e7","Type":"ContainerDied","Data":"4a5ee637ce736b3b554a77f7c72e192a0743b89a63d9a4f07d4a89941ef4d678"} Nov 02 01:51:43.929576 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.927267 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" event={"ID":"93a422c9-9b80-4152-a286-ff6710ec21e7","Type":"ContainerDied","Data":"335cbd38e3c4d1c5f48783f8d400a66aa3f6c9dfb0876829c674cc32840d46fb"} Nov 02 01:51:43.929576 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.927307 5044 scope.go:122] "RemoveContainer" containerID="4a5ee637ce736b3b554a77f7c72e192a0743b89a63d9a4f07d4a89941ef4d678" Nov 02 01:51:43.929576 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.927565 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w" Nov 02 01:51:43.962990 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.962957 5044 scope.go:122] "RemoveContainer" containerID="4a5ee637ce736b3b554a77f7c72e192a0743b89a63d9a4f07d4a89941ef4d678" Nov 02 01:51:43.963403 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:51:43.963375 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a5ee637ce736b3b554a77f7c72e192a0743b89a63d9a4f07d4a89941ef4d678\": container with ID starting with 4a5ee637ce736b3b554a77f7c72e192a0743b89a63d9a4f07d4a89941ef4d678 not found: ID does not exist" containerID="4a5ee637ce736b3b554a77f7c72e192a0743b89a63d9a4f07d4a89941ef4d678" Nov 02 01:51:43.963504 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:43.963415 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a5ee637ce736b3b554a77f7c72e192a0743b89a63d9a4f07d4a89941ef4d678"} err="failed to get container status \"4a5ee637ce736b3b554a77f7c72e192a0743b89a63d9a4f07d4a89941ef4d678\": rpc error: code = NotFound desc = could not find container \"4a5ee637ce736b3b554a77f7c72e192a0743b89a63d9a4f07d4a89941ef4d678\": container with ID starting with 4a5ee637ce736b3b554a77f7c72e192a0743b89a63d9a4f07d4a89941ef4d678 not found: ID does not exist" Nov 02 01:51:44.111807 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:44.111754 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w"] Nov 02 01:51:44.115119 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:44.115089 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-59d70a90-669bb74cf6-jc98w"] Nov 02 01:51:44.673967 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:44.673167 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:45.082020 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:45.081936 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:45.431523 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:45.431436 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:45.457412 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:45.457386 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a422c9-9b80-4152-a286-ff6710ec21e7" path="/var/lib/kubelet/pods/93a422c9-9b80-4152-a286-ff6710ec21e7/volumes" Nov 02 01:51:45.954069 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:45.954037 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm"] Nov 02 01:51:45.954597 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:45.954263 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" podUID="40c0608c-9731-45bf-996a-c21f0fe4e54b" containerName="maas-api" containerID="cri-o://cf2b2b2ddc089a972eb6d0d1063edb15aba97c86926180dcb34ef6c03e49e346" gracePeriod=30 Nov 02 01:51:45.977056 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:45.977029 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp"] Nov 02 01:51:45.977638 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:45.977519 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" podUID="8cf135c0-f351-450a-99f0-b2120842b6cd" containerName="istio-proxy" containerID="cri-o://1c64f391038f1e370e76548554f1a50f92dac017a87fe13aeeaa7f122b14ff89" gracePeriod=30 Nov 02 01:51:45.996106 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:45.996078 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:46.002018 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.001992 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf"] Nov 02 01:51:46.003217 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.003192 5044 cpu_manager.go:414] "containerMap: removing container" podUID="93a422c9-9b80-4152-a286-ff6710ec21e7" containerName="maas-api" Nov 02 01:51:46.003356 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.003341 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a422c9-9b80-4152-a286-ff6710ec21e7" containerName="maas-api" Nov 02 01:51:46.003650 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.003634 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a422c9-9b80-4152-a286-ff6710ec21e7" containerName="maas-api" Nov 02 01:51:46.009516 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.008609 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.059326 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.059297 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf"] Nov 02 01:51:46.071836 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.071803 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.071977 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.071909 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.072044 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.071983 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.072044 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.072032 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.072155 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.072076 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9j4v\" (UniqueName: \"kubernetes.io/projected/7b500c78-94b3-46ff-a95f-abdc414c5f3b-kube-api-access-j9j4v\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.072155 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.072102 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.072310 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.072161 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.072310 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.072242 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.072310 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.072302 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.191742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.173727 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.191742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.173815 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.191742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.173975 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.191742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.174061 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.191742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.174129 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.191742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.174176 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.191742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.174226 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9j4v\" (UniqueName: \"kubernetes.io/projected/7b500c78-94b3-46ff-a95f-abdc414c5f3b-kube-api-access-j9j4v\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.191742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.174263 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.191742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.174321 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.191742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.183419 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.232388 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.207439 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.232388 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.208915 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.232388 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.209787 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.252696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.234997 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.257536 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.257451 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.283197 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.283075 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:46.291533 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.290206 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.292592 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.292526 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9j4v\" (UniqueName: \"kubernetes.io/projected/7b500c78-94b3-46ff-a95f-abdc414c5f3b-kube-api-access-j9j4v\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.308955 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.301301 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rhjxf\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.373693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.348152 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:46.447541 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.447513 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" Nov 02 01:51:46.470804 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.470771 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:46.596809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.594470 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c0608c-9731-45bf-996a-c21f0fe4e54b-kube-api-access-mbp6w\" (UniqueName: \"kubernetes.io/projected/40c0608c-9731-45bf-996a-c21f0fe4e54b-kube-api-access-mbp6w\") pod \"40c0608c-9731-45bf-996a-c21f0fe4e54b\" (UID: \"40c0608c-9731-45bf-996a-c21f0fe4e54b\") " Nov 02 01:51:46.596809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.594625 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40c0608c-9731-45bf-996a-c21f0fe4e54b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40c0608c-9731-45bf-996a-c21f0fe4e54b-maas-api-tls\") pod \"40c0608c-9731-45bf-996a-c21f0fe4e54b\" (UID: \"40c0608c-9731-45bf-996a-c21f0fe4e54b\") " Nov 02 01:51:46.596809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.594710 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c0608c-9731-45bf-996a-c21f0fe4e54b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40c0608c-9731-45bf-996a-c21f0fe4e54b-openshift-service-ca\") pod \"40c0608c-9731-45bf-996a-c21f0fe4e54b\" (UID: \"40c0608c-9731-45bf-996a-c21f0fe4e54b\") " Nov 02 01:51:46.596809 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.594959 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40c0608c-9731-45bf-996a-c21f0fe4e54b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/40c0608c-9731-45bf-996a-c21f0fe4e54b-metrics-certs\") pod \"40c0608c-9731-45bf-996a-c21f0fe4e54b\" (UID: \"40c0608c-9731-45bf-996a-c21f0fe4e54b\") " Nov 02 01:51:46.600691 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.598630 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c0608c-9731-45bf-996a-c21f0fe4e54b-openshift-service-ca" pod "40c0608c-9731-45bf-996a-c21f0fe4e54b" (UID: "40c0608c-9731-45bf-996a-c21f0fe4e54b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:51:46.600691 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.599692 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c0608c-9731-45bf-996a-c21f0fe4e54b-maas-api-tls" pod "40c0608c-9731-45bf-996a-c21f0fe4e54b" (UID: "40c0608c-9731-45bf-996a-c21f0fe4e54b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:51:46.604891 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.604740 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c0608c-9731-45bf-996a-c21f0fe4e54b-metrics-certs" pod "40c0608c-9731-45bf-996a-c21f0fe4e54b" (UID: "40c0608c-9731-45bf-996a-c21f0fe4e54b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:51:46.619605 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.618179 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c0608c-9731-45bf-996a-c21f0fe4e54b-kube-api-access-mbp6w" pod "40c0608c-9731-45bf-996a-c21f0fe4e54b" (UID: "40c0608c-9731-45bf-996a-c21f0fe4e54b"). InnerVolumeSpecName "kube-api-access-mbp6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:51:46.643341 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.643208 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:46.698102 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.698067 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbp6w\" (UniqueName: \"kubernetes.io/projected/40c0608c-9731-45bf-996a-c21f0fe4e54b-kube-api-access-mbp6w\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:46.698102 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.698095 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40c0608c-9731-45bf-996a-c21f0fe4e54b-maas-api-tls\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:46.698338 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.698111 5044 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40c0608c-9731-45bf-996a-c21f0fe4e54b-openshift-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:46.698338 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.698124 5044 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40c0608c-9731-45bf-996a-c21f0fe4e54b-metrics-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:46.810836 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.810797 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:46.882544 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:51:46.882498 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b500c78_94b3_46ff_a95f_abdc414c5f3b.slice/crio-debf0af021d9bc943c8ed5a9228891a2a52771c0f3cec4a1650b931d7dd38410 WatchSource:0}: Error finding container debf0af021d9bc943c8ed5a9228891a2a52771c0f3cec4a1650b931d7dd38410: Status 404 returned error can't find the container with id debf0af021d9bc943c8ed5a9228891a2a52771c0f3cec4a1650b931d7dd38410 Nov 02 01:51:46.884559 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.884531 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf"] Nov 02 01:51:46.885315 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.885279 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:51:46.885421 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.885360 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:51:46.885421 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.885406 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:51:46.956553 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.956526 5044 generic.go:356] "Generic (PLEG): container finished" podID="40c0608c-9731-45bf-996a-c21f0fe4e54b" containerID="cf2b2b2ddc089a972eb6d0d1063edb15aba97c86926180dcb34ef6c03e49e346" exitCode=0 Nov 02 01:51:46.956972 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.956575 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" Nov 02 01:51:46.956972 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.956604 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" event={"ID":"40c0608c-9731-45bf-996a-c21f0fe4e54b","Type":"ContainerDied","Data":"cf2b2b2ddc089a972eb6d0d1063edb15aba97c86926180dcb34ef6c03e49e346"} Nov 02 01:51:46.956972 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.956680 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm" event={"ID":"40c0608c-9731-45bf-996a-c21f0fe4e54b","Type":"ContainerDied","Data":"69ea016a901b3d9e7544f7631ac0980e659de28ef3a4fe10cfc1492d3cc9c89a"} Nov 02 01:51:46.956972 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.956716 5044 scope.go:122] "RemoveContainer" containerID="cf2b2b2ddc089a972eb6d0d1063edb15aba97c86926180dcb34ef6c03e49e346" Nov 02 01:51:46.967693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:46.964825 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" event={"ID":"7b500c78-94b3-46ff-a95f-abdc414c5f3b","Type":"ContainerStarted","Data":"debf0af021d9bc943c8ed5a9228891a2a52771c0f3cec4a1650b931d7dd38410"} Nov 02 01:51:47.147336 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:47.147272 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:47.159705 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:47.156219 5044 scope.go:122] "RemoveContainer" containerID="cf2b2b2ddc089a972eb6d0d1063edb15aba97c86926180dcb34ef6c03e49e346" Nov 02 01:51:47.179734 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:47.160871 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm"] Nov 02 01:51:47.179734 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:51:47.161178 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf2b2b2ddc089a972eb6d0d1063edb15aba97c86926180dcb34ef6c03e49e346\": container with ID starting with cf2b2b2ddc089a972eb6d0d1063edb15aba97c86926180dcb34ef6c03e49e346 not found: ID does not exist" containerID="cf2b2b2ddc089a972eb6d0d1063edb15aba97c86926180dcb34ef6c03e49e346" Nov 02 01:51:47.179734 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:47.161216 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf2b2b2ddc089a972eb6d0d1063edb15aba97c86926180dcb34ef6c03e49e346"} err="failed to get container status \"cf2b2b2ddc089a972eb6d0d1063edb15aba97c86926180dcb34ef6c03e49e346\": rpc error: code = NotFound desc = could not find container \"cf2b2b2ddc089a972eb6d0d1063edb15aba97c86926180dcb34ef6c03e49e346\": container with ID starting with cf2b2b2ddc089a972eb6d0d1063edb15aba97c86926180dcb34ef6c03e49e346 not found: ID does not exist" Nov 02 01:51:47.179734 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:47.164986 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fd723454-56cf47cb65-t8xrm"] Nov 02 01:51:47.455882 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:47.455801 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c0608c-9731-45bf-996a-c21f0fe4e54b" path="/var/lib/kubelet/pods/40c0608c-9731-45bf-996a-c21f0fe4e54b/volumes" Nov 02 01:51:47.711746 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:47.702021 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:47.908578 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:47.908516 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:47.989777 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:47.984027 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" event={"ID":"7b500c78-94b3-46ff-a95f-abdc414c5f3b","Type":"ContainerStarted","Data":"4c692abd1cdf223ce8890af6517f2cb3e27de5a6567954233680a8b8d1d65999"} Nov 02 01:51:48.080698 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:48.080592 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:48.185694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:48.181558 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" podStartSLOduration=3.181535792 podStartE2EDuration="3.181535792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:51:48.164931686 +0000 UTC m=+1624.836184380" watchObservedRunningTime="2026-11-02 01:51:48.181535792 +0000 UTC m=+1624.852788474" Nov 02 01:51:48.331523 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:48.329934 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:48.354752 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:48.353370 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:48.358111 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:48.357896 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 01:51:48.358111 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:48.357978 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" podUID="7b500c78-94b3-46ff-a95f-abdc414c5f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 01:51:48.723028 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:48.722941 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:48.908701 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:48.908369 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:49.114499 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:49.114456 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:49.351164 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:49.351133 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 01:51:49.351347 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:49.351194 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" podUID="7b500c78-94b3-46ff-a95f-abdc414c5f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 01:51:49.816690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:49.816328 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:49.988693 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:49.980312 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:50.124497 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:50.124409 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:50.350877 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:50.350847 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 01:51:50.351056 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:50.350895 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" podUID="7b500c78-94b3-46ff-a95f-abdc414c5f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 01:51:50.722758 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:50.722718 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:51.351177 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.351142 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 01:51:51.351591 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.351207 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" podUID="7b500c78-94b3-46ff-a95f-abdc414c5f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 01:51:51.357774 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.357753 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:51:51.442394 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.442363 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cf135c0-f351-450a-99f0-b2120842b6cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf135c0-f351-450a-99f0-b2120842b6cd-istiod-ca-cert\") pod \"8cf135c0-f351-450a-99f0-b2120842b6cd\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " Nov 02 01:51:51.442558 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.442503 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-credential-socket\") pod \"8cf135c0-f351-450a-99f0-b2120842b6cd\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " Nov 02 01:51:51.442558 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.442531 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-podinfo\") pod \"8cf135c0-f351-450a-99f0-b2120842b6cd\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " Nov 02 01:51:51.442558 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.442552 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf135c0-f351-450a-99f0-b2120842b6cd-kube-api-access-zssmn\" (UniqueName: \"kubernetes.io/projected/8cf135c0-f351-450a-99f0-b2120842b6cd-kube-api-access-zssmn\") pod \"8cf135c0-f351-450a-99f0-b2120842b6cd\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " Nov 02 01:51:51.442657 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.442599 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-data\") pod \"8cf135c0-f351-450a-99f0-b2120842b6cd\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " Nov 02 01:51:51.442797 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.442775 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-envoy\") pod \"8cf135c0-f351-450a-99f0-b2120842b6cd\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " Nov 02 01:51:51.442885 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.442855 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-workload-certs\") pod \"8cf135c0-f351-450a-99f0-b2120842b6cd\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " Nov 02 01:51:51.442948 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.442903 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-credential-socket" pod "8cf135c0-f351-450a-99f0-b2120842b6cd" (UID: "8cf135c0-f351-450a-99f0-b2120842b6cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:51:51.442948 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.442915 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf135c0-f351-450a-99f0-b2120842b6cd-istiod-ca-cert" pod "8cf135c0-f351-450a-99f0-b2120842b6cd" (UID: "8cf135c0-f351-450a-99f0-b2120842b6cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:51:51.442948 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.442927 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-workload-socket\") pod \"8cf135c0-f351-450a-99f0-b2120842b6cd\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " Nov 02 01:51:51.442948 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.442939 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-data" pod "8cf135c0-f351-450a-99f0-b2120842b6cd" (UID: "8cf135c0-f351-450a-99f0-b2120842b6cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:51:51.443174 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.442995 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-token\" (UniqueName: \"kubernetes.io/projected/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-token\") pod \"8cf135c0-f351-450a-99f0-b2120842b6cd\" (UID: \"8cf135c0-f351-450a-99f0-b2120842b6cd\") " Nov 02 01:51:51.443174 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.443105 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-workload-certs" pod "8cf135c0-f351-450a-99f0-b2120842b6cd" (UID: "8cf135c0-f351-450a-99f0-b2120842b6cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:51:51.443272 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.443197 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-workload-socket" pod "8cf135c0-f351-450a-99f0-b2120842b6cd" (UID: "8cf135c0-f351-450a-99f0-b2120842b6cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:51:51.443773 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.443744 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf135c0-f351-450a-99f0-b2120842b6cd-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:51.443904 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.443776 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:51.443904 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.443792 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:51.443904 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.443806 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:51.443904 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.443826 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:51.445318 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.445212 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-podinfo" pod "8cf135c0-f351-450a-99f0-b2120842b6cd" (UID: "8cf135c0-f351-450a-99f0-b2120842b6cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:51:51.445318 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.445270 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf135c0-f351-450a-99f0-b2120842b6cd-kube-api-access-zssmn" pod "8cf135c0-f351-450a-99f0-b2120842b6cd" (UID: "8cf135c0-f351-450a-99f0-b2120842b6cd"). InnerVolumeSpecName "kube-api-access-zssmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:51:51.445620 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.445598 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-envoy" pod "8cf135c0-f351-450a-99f0-b2120842b6cd" (UID: "8cf135c0-f351-450a-99f0-b2120842b6cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:51:51.445759 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.445739 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-token" pod "8cf135c0-f351-450a-99f0-b2120842b6cd" (UID: "8cf135c0-f351-450a-99f0-b2120842b6cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:51:51.546262 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.546228 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:51.546262 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.546263 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zssmn\" (UniqueName: \"kubernetes.io/projected/8cf135c0-f351-450a-99f0-b2120842b6cd-kube-api-access-zssmn\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:51.546494 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.546279 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:51.546494 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:51.546295 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf135c0-f351-450a-99f0-b2120842b6cd-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:52.018681 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:52.018631 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:52.067768 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:52.067729 5044 generic.go:356] "Generic (PLEG): container finished" podID="8cf135c0-f351-450a-99f0-b2120842b6cd" containerID="1c64f391038f1e370e76548554f1a50f92dac017a87fe13aeeaa7f122b14ff89" exitCode=0 Nov 02 01:51:52.068192 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:52.068163 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" event={"ID":"8cf135c0-f351-450a-99f0-b2120842b6cd","Type":"ContainerDied","Data":"1c64f391038f1e370e76548554f1a50f92dac017a87fe13aeeaa7f122b14ff89"} Nov 02 01:51:52.068408 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:52.068393 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" event={"ID":"8cf135c0-f351-450a-99f0-b2120842b6cd","Type":"ContainerDied","Data":"0f0ffc14a2a938bbd67eb77dae58b5a777b46c3a1d44212e5de0026fee412d45"} Nov 02 01:51:52.068545 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:52.068532 5044 scope.go:122] "RemoveContainer" containerID="1c64f391038f1e370e76548554f1a50f92dac017a87fe13aeeaa7f122b14ff89" Nov 02 01:51:52.068712 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:52.068319 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp" Nov 02 01:51:52.102980 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:52.102958 5044 scope.go:122] "RemoveContainer" containerID="d6e47f3076aebabad242bf6a12361bad610932c2867f4741b90e8e8659766faa" Nov 02 01:51:52.114169 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:52.114025 5044 scope.go:122] "RemoveContainer" containerID="1c64f391038f1e370e76548554f1a50f92dac017a87fe13aeeaa7f122b14ff89" Nov 02 01:51:52.114328 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:51:52.114308 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c64f391038f1e370e76548554f1a50f92dac017a87fe13aeeaa7f122b14ff89\": container with ID starting with 1c64f391038f1e370e76548554f1a50f92dac017a87fe13aeeaa7f122b14ff89 not found: ID does not exist" containerID="1c64f391038f1e370e76548554f1a50f92dac017a87fe13aeeaa7f122b14ff89" Nov 02 01:51:52.114408 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:52.114341 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c64f391038f1e370e76548554f1a50f92dac017a87fe13aeeaa7f122b14ff89"} err="failed to get container status \"1c64f391038f1e370e76548554f1a50f92dac017a87fe13aeeaa7f122b14ff89\": rpc error: code = NotFound desc = could not find container \"1c64f391038f1e370e76548554f1a50f92dac017a87fe13aeeaa7f122b14ff89\": container with ID starting with 1c64f391038f1e370e76548554f1a50f92dac017a87fe13aeeaa7f122b14ff89 not found: ID does not exist" Nov 02 01:51:52.114408 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:52.114367 5044 scope.go:122] "RemoveContainer" containerID="d6e47f3076aebabad242bf6a12361bad610932c2867f4741b90e8e8659766faa" Nov 02 01:51:52.114646 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:51:52.114620 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6e47f3076aebabad242bf6a12361bad610932c2867f4741b90e8e8659766faa\": container with ID starting with d6e47f3076aebabad242bf6a12361bad610932c2867f4741b90e8e8659766faa not found: ID does not exist" containerID="d6e47f3076aebabad242bf6a12361bad610932c2867f4741b90e8e8659766faa" Nov 02 01:51:52.114738 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:52.114656 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6e47f3076aebabad242bf6a12361bad610932c2867f4741b90e8e8659766faa"} err="failed to get container status \"d6e47f3076aebabad242bf6a12361bad610932c2867f4741b90e8e8659766faa\": rpc error: code = NotFound desc = could not find container \"d6e47f3076aebabad242bf6a12361bad610932c2867f4741b90e8e8659766faa\": container with ID starting with d6e47f3076aebabad242bf6a12361bad610932c2867f4741b90e8e8659766faa not found: ID does not exist" Nov 02 01:51:52.195300 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:52.194258 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:52.243840 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:52.243807 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp"] Nov 02 01:51:52.299653 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:52.299523 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-59d70a90-openshift-default-747bc44b95-xn7kp"] Nov 02 01:51:52.350166 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:52.350136 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 01:51:52.350329 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:52.350195 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" podUID="7b500c78-94b3-46ff-a95f-abdc414c5f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 01:51:53.213972 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:53.213932 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9"] Nov 02 01:51:53.214489 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:53.214186 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" podUID="01a827d2-d8ce-4cc8-b440-c171c094c1da" containerName="maas-api" containerID="cri-o://0fcc3b5233e96270385ff059a6029a4dd27818f46462ed1622701b2bae7612d9" gracePeriod=30 Nov 02 01:51:53.315980 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:53.315945 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:53.351205 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:53.351166 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 01:51:53.351341 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:53.351251 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" podUID="7b500c78-94b3-46ff-a95f-abdc414c5f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 01:51:53.497015 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:53.488597 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cf135c0-f351-450a-99f0-b2120842b6cd" path="/var/lib/kubelet/pods/8cf135c0-f351-450a-99f0-b2120842b6cd/volumes" Nov 02 01:51:53.879690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:53.877807 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" Nov 02 01:51:53.959543 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:53.959491 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01a827d2-d8ce-4cc8-b440-c171c094c1da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01a827d2-d8ce-4cc8-b440-c171c094c1da-metrics-certs\") pod \"01a827d2-d8ce-4cc8-b440-c171c094c1da\" (UID: \"01a827d2-d8ce-4cc8-b440-c171c094c1da\") " Nov 02 01:51:53.959783 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:53.959615 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a827d2-d8ce-4cc8-b440-c171c094c1da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a827d2-d8ce-4cc8-b440-c171c094c1da-openshift-service-ca\") pod \"01a827d2-d8ce-4cc8-b440-c171c094c1da\" (UID: \"01a827d2-d8ce-4cc8-b440-c171c094c1da\") " Nov 02 01:51:53.959783 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:53.959704 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a827d2-d8ce-4cc8-b440-c171c094c1da-kube-api-access-2s75p\" (UniqueName: \"kubernetes.io/projected/01a827d2-d8ce-4cc8-b440-c171c094c1da-kube-api-access-2s75p\") pod \"01a827d2-d8ce-4cc8-b440-c171c094c1da\" (UID: \"01a827d2-d8ce-4cc8-b440-c171c094c1da\") " Nov 02 01:51:53.959783 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:53.959740 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01a827d2-d8ce-4cc8-b440-c171c094c1da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a827d2-d8ce-4cc8-b440-c171c094c1da-maas-api-tls\") pod \"01a827d2-d8ce-4cc8-b440-c171c094c1da\" (UID: \"01a827d2-d8ce-4cc8-b440-c171c094c1da\") " Nov 02 01:51:53.960592 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:53.960556 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a827d2-d8ce-4cc8-b440-c171c094c1da-openshift-service-ca" pod "01a827d2-d8ce-4cc8-b440-c171c094c1da" (UID: "01a827d2-d8ce-4cc8-b440-c171c094c1da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:51:53.962780 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:53.962740 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a827d2-d8ce-4cc8-b440-c171c094c1da-metrics-certs" pod "01a827d2-d8ce-4cc8-b440-c171c094c1da" (UID: "01a827d2-d8ce-4cc8-b440-c171c094c1da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:51:53.962891 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:53.962784 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a827d2-d8ce-4cc8-b440-c171c094c1da-kube-api-access-2s75p" pod "01a827d2-d8ce-4cc8-b440-c171c094c1da" (UID: "01a827d2-d8ce-4cc8-b440-c171c094c1da"). InnerVolumeSpecName "kube-api-access-2s75p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:51:53.963005 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:53.962985 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a827d2-d8ce-4cc8-b440-c171c094c1da-maas-api-tls" pod "01a827d2-d8ce-4cc8-b440-c171c094c1da" (UID: "01a827d2-d8ce-4cc8-b440-c171c094c1da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:51:54.061855 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:54.061818 5044 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a827d2-d8ce-4cc8-b440-c171c094c1da-openshift-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:54.061855 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:54.061854 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s75p\" (UniqueName: \"kubernetes.io/projected/01a827d2-d8ce-4cc8-b440-c171c094c1da-kube-api-access-2s75p\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:54.062111 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:54.061871 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a827d2-d8ce-4cc8-b440-c171c094c1da-maas-api-tls\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:54.062111 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:54.061884 5044 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01a827d2-d8ce-4cc8-b440-c171c094c1da-metrics-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:51:54.104504 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:54.104408 5044 generic.go:356] "Generic (PLEG): container finished" podID="01a827d2-d8ce-4cc8-b440-c171c094c1da" containerID="0fcc3b5233e96270385ff059a6029a4dd27818f46462ed1622701b2bae7612d9" exitCode=0 Nov 02 01:51:54.104504 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:54.104472 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" Nov 02 01:51:54.104755 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:54.104477 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" event={"ID":"01a827d2-d8ce-4cc8-b440-c171c094c1da","Type":"ContainerDied","Data":"0fcc3b5233e96270385ff059a6029a4dd27818f46462ed1622701b2bae7612d9"} Nov 02 01:51:54.104755 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:54.104626 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9" event={"ID":"01a827d2-d8ce-4cc8-b440-c171c094c1da","Type":"ContainerDied","Data":"feb573327673202a3650bb000748d8f4863881f14df570e1659aecf507834dca"} Nov 02 01:51:54.104755 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:54.104649 5044 scope.go:122] "RemoveContainer" containerID="0fcc3b5233e96270385ff059a6029a4dd27818f46462ed1622701b2bae7612d9" Nov 02 01:51:54.131106 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:54.131082 5044 scope.go:122] "RemoveContainer" containerID="0fcc3b5233e96270385ff059a6029a4dd27818f46462ed1622701b2bae7612d9" Nov 02 01:51:54.131420 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:51:54.131398 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fcc3b5233e96270385ff059a6029a4dd27818f46462ed1622701b2bae7612d9\": container with ID starting with 0fcc3b5233e96270385ff059a6029a4dd27818f46462ed1622701b2bae7612d9 not found: ID does not exist" containerID="0fcc3b5233e96270385ff059a6029a4dd27818f46462ed1622701b2bae7612d9" Nov 02 01:51:54.131524 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:54.131429 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fcc3b5233e96270385ff059a6029a4dd27818f46462ed1622701b2bae7612d9"} err="failed to get container status \"0fcc3b5233e96270385ff059a6029a4dd27818f46462ed1622701b2bae7612d9\": rpc error: code = NotFound desc = could not find container \"0fcc3b5233e96270385ff059a6029a4dd27818f46462ed1622701b2bae7612d9\": container with ID starting with 0fcc3b5233e96270385ff059a6029a4dd27818f46462ed1622701b2bae7612d9 not found: ID does not exist" Nov 02 01:51:54.182278 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:51:54.182233 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:51:54.277502 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:54.277467 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9"] Nov 02 01:51:54.284143 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:54.284116 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5fc56410-7b9cb9f789-sr4s9"] Nov 02 01:51:54.362630 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:54.362068 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 01:51:54.362630 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:54.362134 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" podUID="7b500c78-94b3-46ff-a95f-abdc414c5f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 01:51:55.350686 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:55.350640 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 02 01:51:55.351074 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:55.350738 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" podUID="7b500c78-94b3-46ff-a95f-abdc414c5f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 02 01:51:55.364624 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:55.364595 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:55.462950 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:55.461202 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a827d2-d8ce-4cc8-b440-c171c094c1da" path="/var/lib/kubelet/pods/01a827d2-d8ce-4cc8-b440-c171c094c1da/volumes" Nov 02 01:51:55.512946 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:55.511260 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:55.690662 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:55.690560 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:56.637355 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:56.637309 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:56.638168 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:56.638111 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:56.639002 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:56.638977 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 01:51:57.418655 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:57.418615 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:58.328505 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:58.328473 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:51:58.480173 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:51:58.480143 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:00.511857 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:00.511820 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:01.077997 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.077964 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b"] Nov 02 01:52:01.078785 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.078763 5044 cpu_manager.go:414] "containerMap: removing container" podUID="8cf135c0-f351-450a-99f0-b2120842b6cd" containerName="istio-proxy" Nov 02 01:52:01.078785 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.078783 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf135c0-f351-450a-99f0-b2120842b6cd" containerName="istio-proxy" Nov 02 01:52:01.078911 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.078814 5044 cpu_manager.go:414] "containerMap: removing container" podUID="8cf135c0-f351-450a-99f0-b2120842b6cd" containerName="istio-proxy" Nov 02 01:52:01.078911 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.078819 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf135c0-f351-450a-99f0-b2120842b6cd" containerName="istio-proxy" Nov 02 01:52:01.078911 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.078833 5044 cpu_manager.go:414] "containerMap: removing container" podUID="01a827d2-d8ce-4cc8-b440-c171c094c1da" containerName="maas-api" Nov 02 01:52:01.078911 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.078838 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a827d2-d8ce-4cc8-b440-c171c094c1da" containerName="maas-api" Nov 02 01:52:01.078911 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.078853 5044 cpu_manager.go:414] "containerMap: removing container" podUID="40c0608c-9731-45bf-996a-c21f0fe4e54b" containerName="maas-api" Nov 02 01:52:01.078911 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.078858 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c0608c-9731-45bf-996a-c21f0fe4e54b" containerName="maas-api" Nov 02 01:52:01.079105 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.078978 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c0608c-9731-45bf-996a-c21f0fe4e54b" containerName="maas-api" Nov 02 01:52:01.079105 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.078991 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf135c0-f351-450a-99f0-b2120842b6cd" containerName="istio-proxy" Nov 02 01:52:01.079105 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.079000 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf135c0-f351-450a-99f0-b2120842b6cd" containerName="istio-proxy" Nov 02 01:52:01.079105 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.079007 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a827d2-d8ce-4cc8-b440-c171c094c1da" containerName="maas-api" Nov 02 01:52:01.083873 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.083850 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.231438 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.231400 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.231626 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.231448 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.231626 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.231603 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.231808 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.231786 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5p96\" (UniqueName: \"kubernetes.io/projected/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-kube-api-access-s5p96\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.231871 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.231858 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.231932 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.231916 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.232079 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.232052 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.232143 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.232100 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.232185 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.232158 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.301371 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.301334 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b"] Nov 02 01:52:01.334015 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.333900 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.334015 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.334003 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5p96\" (UniqueName: \"kubernetes.io/projected/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-kube-api-access-s5p96\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.334326 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.334040 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.334326 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.334091 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.335204 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.335176 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.335708 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.335682 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.335808 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.334158 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.335808 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.335796 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.335950 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.335841 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.336033 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.336017 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.336094 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.336065 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.336310 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.336290 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.336472 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.336441 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.336542 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.336515 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.338850 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.338825 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.339076 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.339044 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.397044 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.397018 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5p96\" (UniqueName: \"kubernetes.io/projected/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-kube-api-access-s5p96\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.397183 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.397048 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-qwp4b\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.398873 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.398851 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:01.871951 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:52:01.871920 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766dfcbb_afda_4f73_8b00_8c53e2fb45d5.slice/crio-31445b5342c113648255090c5ea98f3552b51ad8a4e78974a035346780b19011 WatchSource:0}: Error finding container 31445b5342c113648255090c5ea98f3552b51ad8a4e78974a035346780b19011: Status 404 returned error can't find the container with id 31445b5342c113648255090c5ea98f3552b51ad8a4e78974a035346780b19011 Nov 02 01:52:01.872327 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.872134 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b"] Nov 02 01:52:01.874501 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.874469 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:52:01.874591 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.874551 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:52:01.874591 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:01.874582 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:52:02.174459 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:02.174370 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" event={"ID":"766dfcbb-afda-4f73-8b00-8c53e2fb45d5","Type":"ContainerStarted","Data":"fb0067c9c9529e478e01367b5810cc450623aaaac89f0e769248a9c38dcd4fd7"} Nov 02 01:52:02.174459 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:02.174445 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" event={"ID":"766dfcbb-afda-4f73-8b00-8c53e2fb45d5","Type":"ContainerStarted","Data":"31445b5342c113648255090c5ea98f3552b51ad8a4e78974a035346780b19011"} Nov 02 01:52:02.297732 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:02.297666 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" podStartSLOduration=1.297651551 podStartE2EDuration="1.297651551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:52:02.296769093 +0000 UTC m=+1638.968021775" watchObservedRunningTime="2026-11-02 01:52:02.297651551 +0000 UTC m=+1638.968904306" Nov 02 01:52:02.404905 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:02.402998 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:02.405335 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:02.405314 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 01:52:02.405392 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:02.405363 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" podUID="766dfcbb-afda-4f73-8b00-8c53e2fb45d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 01:52:03.400092 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:03.400057 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 01:52:03.400583 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:03.400116 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" podUID="766dfcbb-afda-4f73-8b00-8c53e2fb45d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 01:52:04.399595 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:04.399545 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 01:52:04.399791 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:04.399616 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" podUID="766dfcbb-afda-4f73-8b00-8c53e2fb45d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 01:52:05.399361 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:05.399321 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 01:52:05.399912 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:05.399386 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" podUID="766dfcbb-afda-4f73-8b00-8c53e2fb45d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 01:52:05.587054 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:05.587021 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:06.399731 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:06.399663 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 01:52:06.400223 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:06.399752 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" podUID="766dfcbb-afda-4f73-8b00-8c53e2fb45d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 01:52:07.400203 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:07.400159 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 01:52:07.400761 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:07.400221 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" podUID="766dfcbb-afda-4f73-8b00-8c53e2fb45d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 01:52:08.400079 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:08.400034 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 01:52:08.400251 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:08.400108 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" podUID="766dfcbb-afda-4f73-8b00-8c53e2fb45d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 01:52:09.400028 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:09.399991 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 02 01:52:09.400228 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:09.400074 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" podUID="766dfcbb-afda-4f73-8b00-8c53e2fb45d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 02 01:52:10.405120 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:10.405084 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:10.405545 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:10.405467 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:10.406204 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:10.406179 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 01:52:10.693902 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:10.693787 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:14.553708 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:14.553644 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:14.803818 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:14.803716 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:15.085517 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:15.085429 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:15.487122 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:15.487040 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:17.099985 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:17.099952 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:20.434296 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:20.434247 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:21.118146 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:21.118112 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:21.283318 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:21.283219 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:23.062652 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:23.062556 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:24.178104 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:52:24.178059 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:52:25.479941 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:25.479895 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:30.322123 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:30.322087 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:34.830158 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:34.830124 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6"] Nov 02 01:52:34.834656 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:34.834634 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" Nov 02 01:52:34.910268 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:34.910213 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-metrics-certs\") pod \"maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6\" (UID: \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" Nov 02 01:52:34.910449 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:34.910367 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg7fs\" (UniqueName: \"kubernetes.io/projected/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-kube-api-access-fg7fs\") pod \"maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6\" (UID: \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" Nov 02 01:52:34.910523 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:34.910489 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-maas-api-tls\") pod \"maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6\" (UID: \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" Nov 02 01:52:34.910611 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:34.910581 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-openshift-service-ca\") pod \"maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6\" (UID: \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" Nov 02 01:52:34.938704 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:34.938657 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-8ebd9167-serving-cert\"" Nov 02 01:52:34.943081 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:34.943052 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6"] Nov 02 01:52:35.012107 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:35.012069 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg7fs\" (UniqueName: \"kubernetes.io/projected/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-kube-api-access-fg7fs\") pod \"maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6\" (UID: \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" Nov 02 01:52:35.012286 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:35.012231 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-maas-api-tls\") pod \"maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6\" (UID: \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" Nov 02 01:52:35.012374 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:35.012357 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-openshift-service-ca\") pod \"maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6\" (UID: \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" Nov 02 01:52:35.012497 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:35.012482 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-metrics-certs\") pod \"maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6\" (UID: \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" Nov 02 01:52:35.012554 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:52:35.012512 5044 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-serving-cert: secret "maas-api-e2e-migrate-8ebd9167-serving-cert" not found Nov 02 01:52:35.012612 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:52:35.012593 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-maas-api-tls podName:4ac69c4e-27a7-41b7-a2d5-dfce81e50934 nodeName:}" failed. No retries permitted until 2026-11-02 01:52:35.512571399 +0000 UTC m=+1672.183824080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-maas-api-tls") pod "maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" (UID: "4ac69c4e-27a7-41b7-a2d5-dfce81e50934") : secret "maas-api-e2e-migrate-8ebd9167-serving-cert" not found Nov 02 01:52:35.013497 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:35.013467 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-openshift-service-ca\") pod \"maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6\" (UID: \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" Nov 02 01:52:35.015556 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:35.015527 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-metrics-certs\") pod \"maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6\" (UID: \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" Nov 02 01:52:35.022297 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:35.022258 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:35.136441 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:35.136366 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg7fs\" (UniqueName: \"kubernetes.io/projected/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-kube-api-access-fg7fs\") pod \"maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6\" (UID: \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" Nov 02 01:52:35.188698 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:35.184991 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:35.486907 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:35.486816 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:35.526016 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:35.525957 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-maas-api-tls\") pod \"maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6\" (UID: \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" Nov 02 01:52:35.528484 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:35.528461 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-maas-api-tls\") pod \"maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6\" (UID: \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" Nov 02 01:52:35.750149 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:35.750042 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" Nov 02 01:52:36.139942 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:52:36.139258 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac69c4e_27a7_41b7_a2d5_dfce81e50934.slice/crio-d4584bd1fb7ae734b892f9f9ff1845968adac9af84a0df9c9c5faa0faec3d2a1 WatchSource:0}: Error finding container d4584bd1fb7ae734b892f9f9ff1845968adac9af84a0df9c9c5faa0faec3d2a1: Status 404 returned error can't find the container with id d4584bd1fb7ae734b892f9f9ff1845968adac9af84a0df9c9c5faa0faec3d2a1 Nov 02 01:52:36.140367 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:36.140259 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6"] Nov 02 01:52:36.500256 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:36.500179 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" event={"ID":"4ac69c4e-27a7-41b7-a2d5-dfce81e50934","Type":"ContainerStarted","Data":"d4584bd1fb7ae734b892f9f9ff1845968adac9af84a0df9c9c5faa0faec3d2a1"} Nov 02 01:52:38.521469 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:38.521424 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" event={"ID":"4ac69c4e-27a7-41b7-a2d5-dfce81e50934","Type":"ContainerStarted","Data":"591b213b450e4a7b51a96f5b8a7bf0cc38b82be0b31f80cb5c8e3435868bf1b9"} Nov 02 01:52:38.522007 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:38.521528 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" Nov 02 01:52:38.646710 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:38.646640 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" podStartSLOduration=3.289520539 podStartE2EDuration="4.646626498s" totalImagesPullingTime="1.357105959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:52:38.642084712 +0000 UTC m=+1675.313337394" watchObservedRunningTime="2026-11-02 01:52:38.646626498 +0000 UTC m=+1675.317879180" Nov 02 01:52:39.788959 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:39.788925 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:40.412022 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:40.411989 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:40.615543 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:40.615206 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:41.274085 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:41.274004 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:42.403850 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:42.403818 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:42.559141 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:42.559104 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:42.756749 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:42.756654 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:44.537280 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:44.537232 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" Nov 02 01:52:45.287664 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:45.287624 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:45.438109 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:45.438066 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:45.582614 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:45.582525 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:47.166180 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:47.166150 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:47.325326 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:47.325290 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:47.470292 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:47.470205 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:50.507983 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:50.507942 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:54.176138 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:52:54.176096 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:52:55.381117 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:55.381085 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:52:58.053404 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:58.052340 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf"] Nov 02 01:52:58.100901 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:58.100866 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ab93b6-mj2zf"] Nov 02 01:52:59.457001 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:52:59.456956 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e040acd1-c6d0-43bd-90fb-29bdc08f3a3a" path="/var/lib/kubelet/pods/e040acd1-c6d0-43bd-90fb-29bdc08f3a3a/volumes" Nov 02 01:53:00.228787 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:00.228749 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:02.051617 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:02.051576 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc"] Nov 02 01:53:02.099088 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:02.099056 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad04b64-stqmc"] Nov 02 01:53:03.454110 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:03.454076 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c631f891-1a57-469f-ba37-00f34847792e" path="/var/lib/kubelet/pods/c631f891-1a57-469f-ba37-00f34847792e/volumes" Nov 02 01:53:05.075127 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:05.075092 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:09.793245 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:09.793203 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:14.719247 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:14.719212 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:15.751075 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:15.751043 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:16.422895 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:16.422848 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:16.803343 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:16.803305 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:19.575430 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:19.575399 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:21.424742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:21.424700 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:21.571217 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:21.571176 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:21.715181 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:21.715085 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:24.176518 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:53:24.176483 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:53:24.465067 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:24.464979 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:27.054194 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:27.054159 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr"] Nov 02 01:53:27.104922 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:27.104889 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c41b753-v6gxr"] Nov 02 01:53:27.455117 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:27.455036 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0d78e2c-221f-47f2-966f-bd42c211bae9" path="/var/lib/kubelet/pods/d0d78e2c-221f-47f2-966f-bd42c211bae9/volumes" Nov 02 01:53:29.664587 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:29.664549 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:29.829991 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:29.829955 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:29.992264 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:29.992170 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:35.048050 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:35.048018 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:35.759875 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:35.759845 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:35.946945 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:35.946896 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:39.050288 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:39.050240 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm"] Nov 02 01:53:39.100906 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:39.100870 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-019f3a91-gdgcm"] Nov 02 01:53:39.454259 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:39.454176 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84dd5a1d-8f13-49f7-9d97-16952e86ee5f" path="/var/lib/kubelet/pods/84dd5a1d-8f13-49f7-9d97-16952e86ee5f/volumes" Nov 02 01:53:40.072910 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:40.072875 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:40.866243 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:40.866213 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:41.008091 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:41.008055 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:42.812806 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:42.812764 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:44.964113 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:44.964076 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:45.964173 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:45.964135 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:46.116961 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:46.116923 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:46.275656 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:46.275625 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:46.621137 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:46.621052 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:46.818563 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:46.818510 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:47.566509 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:47.566480 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:47.771167 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:47.771132 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:47.919363 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:47.919156 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:48.483717 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:48.477905 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:48.855730 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:48.855636 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:49.856630 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:49.856594 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:50.459153 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:50.459121 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:50.660932 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:50.660898 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:50.815793 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:50.815761 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:53.995933 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:53.995862 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:54.175714 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:53:54.175660 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:53:54.184632 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:54.184600 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:54.331108 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:54.331079 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:55.868379 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:55.868346 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:56.347007 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:56.346977 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:53:58.769196 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:58.769163 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg"] Nov 02 01:53:58.769747 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:58.769452 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" podUID="e541a05c-0d9f-420f-af63-638e2642c6aa" containerName="istio-proxy" containerID="cri-o://04d69b4c5a6c8affc1a365226c709c8731d992c16f31e32bc2de72122fdf62bb" gracePeriod=30 Nov 02 01:53:58.819296 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:58.819261 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k"] Nov 02 01:53:58.819823 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:58.819793 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" podUID="1dbf2ea9-cb23-4fa2-b246-58ca09066b74" containerName="istio-proxy" containerID="cri-o://d907db85be7e47d9345985af951de423ee97fdec394c1640192ca41bdaffa41f" gracePeriod=30 Nov 02 01:53:58.825085 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:58.825058 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp"] Nov 02 01:53:58.825471 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:58.825442 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" podUID="ce5a9bbf-1908-4220-b871-452bac6bebb8" containerName="istio-proxy" containerID="cri-o://de1c9b12ae2050bdc1a6d7db5c9da8477ae71fd00d506495d5c1e7885f40cc1e" gracePeriod=30 Nov 02 01:53:58.870122 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:58.870039 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284"] Nov 02 01:53:58.870265 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:53:58.870224 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" podUID="2d7365ed-68db-449b-9ef1-047cc14f55c3" containerName="istio-proxy" containerID="cri-o://86412078dfd576c857e7ae73bc01d9684ce0c333c38094343c81e3ce1ea6ea23" gracePeriod=30 Nov 02 01:54:00.700340 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:00.700297 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:54:00.700749 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:00.700372 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" podUID="ce5a9bbf-1908-4220-b871-452bac6bebb8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:54:01.183939 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:01.183904 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:03.757009 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:03.756969 5044 patch_prober.go:29] interesting pod/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:54:03.757389 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:03.757042 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" podUID="e541a05c-0d9f-420f-af63-638e2642c6aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:54:04.144008 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.143976 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:54:04.154109 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.154077 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-token\" (UniqueName: \"kubernetes.io/projected/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-token\") pod \"e541a05c-0d9f-420f-af63-638e2642c6aa\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " Nov 02 01:54:04.154245 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.154213 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-workload-socket\") pod \"e541a05c-0d9f-420f-af63-638e2642c6aa\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " Nov 02 01:54:04.154326 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.154253 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e541a05c-0d9f-420f-af63-638e2642c6aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e541a05c-0d9f-420f-af63-638e2642c6aa-istiod-ca-cert\") pod \"e541a05c-0d9f-420f-af63-638e2642c6aa\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " Nov 02 01:54:04.154391 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.154360 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e541a05c-0d9f-420f-af63-638e2642c6aa-kube-api-access-jfbjr\" (UniqueName: \"kubernetes.io/projected/e541a05c-0d9f-420f-af63-638e2642c6aa-kube-api-access-jfbjr\") pod \"e541a05c-0d9f-420f-af63-638e2642c6aa\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " Nov 02 01:54:04.154479 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.154442 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-data\") pod \"e541a05c-0d9f-420f-af63-638e2642c6aa\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " Nov 02 01:54:04.154538 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.154511 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-podinfo\") pod \"e541a05c-0d9f-420f-af63-638e2642c6aa\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " Nov 02 01:54:04.154593 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.154563 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-credential-socket\") pod \"e541a05c-0d9f-420f-af63-638e2642c6aa\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " Nov 02 01:54:04.154648 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.154595 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-envoy\") pod \"e541a05c-0d9f-420f-af63-638e2642c6aa\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " Nov 02 01:54:04.154987 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.154743 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-workload-certs\") pod \"e541a05c-0d9f-420f-af63-638e2642c6aa\" (UID: \"e541a05c-0d9f-420f-af63-638e2642c6aa\") " Nov 02 01:54:04.155330 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.155132 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-data" pod "e541a05c-0d9f-420f-af63-638e2642c6aa" (UID: "e541a05c-0d9f-420f-af63-638e2642c6aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:04.155931 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.155708 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.155931 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.155900 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e541a05c-0d9f-420f-af63-638e2642c6aa-istiod-ca-cert" pod "e541a05c-0d9f-420f-af63-638e2642c6aa" (UID: "e541a05c-0d9f-420f-af63-638e2642c6aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:54:04.156288 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.156263 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-workload-socket" pod "e541a05c-0d9f-420f-af63-638e2642c6aa" (UID: "e541a05c-0d9f-420f-af63-638e2642c6aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:04.156596 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.156554 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-credential-socket" pod "e541a05c-0d9f-420f-af63-638e2642c6aa" (UID: "e541a05c-0d9f-420f-af63-638e2642c6aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:04.156717 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.156651 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-workload-certs" pod "e541a05c-0d9f-420f-af63-638e2642c6aa" (UID: "e541a05c-0d9f-420f-af63-638e2642c6aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:04.157041 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.157013 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-token" pod "e541a05c-0d9f-420f-af63-638e2642c6aa" (UID: "e541a05c-0d9f-420f-af63-638e2642c6aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:54:04.160632 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.160595 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e541a05c-0d9f-420f-af63-638e2642c6aa-kube-api-access-jfbjr" pod "e541a05c-0d9f-420f-af63-638e2642c6aa" (UID: "e541a05c-0d9f-420f-af63-638e2642c6aa"). InnerVolumeSpecName "kube-api-access-jfbjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:54:04.163161 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.163130 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-envoy" pod "e541a05c-0d9f-420f-af63-638e2642c6aa" (UID: "e541a05c-0d9f-420f-af63-638e2642c6aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:04.166095 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.166062 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-podinfo" pod "e541a05c-0d9f-420f-af63-638e2642c6aa" (UID: "e541a05c-0d9f-420f-af63-638e2642c6aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:54:04.261438 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.261403 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.261586 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.261442 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.261586 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.261457 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.261586 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.261469 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e541a05c-0d9f-420f-af63-638e2642c6aa-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.261586 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.261482 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfbjr\" (UniqueName: \"kubernetes.io/projected/e541a05c-0d9f-420f-af63-638e2642c6aa-kube-api-access-jfbjr\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.261586 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.261496 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.261586 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.261510 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.261586 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.261523 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e541a05c-0d9f-420f-af63-638e2642c6aa-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.269750 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.269722 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:54:04.316636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.316533 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:54:04.334691 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.334130 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:54:04.362223 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.362147 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-workload-socket\") pod \"2d7365ed-68db-449b-9ef1-047cc14f55c3\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " Nov 02 01:54:04.362356 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.362243 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-credential-socket\") pod \"2d7365ed-68db-449b-9ef1-047cc14f55c3\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " Nov 02 01:54:04.362356 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.362308 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-token\" (UniqueName: \"kubernetes.io/projected/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-token\") pod \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " Nov 02 01:54:04.362356 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.362343 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-workload-certs\") pod \"ce5a9bbf-1908-4220-b871-452bac6bebb8\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " Nov 02 01:54:04.362476 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.362371 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-token\" (UniqueName: \"kubernetes.io/projected/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-token\") pod \"2d7365ed-68db-449b-9ef1-047cc14f55c3\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " Nov 02 01:54:04.362476 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.362456 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-credential-socket\") pod \"ce5a9bbf-1908-4220-b871-452bac6bebb8\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " Nov 02 01:54:04.362547 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.362487 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-data\") pod \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " Nov 02 01:54:04.362547 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.362519 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-data\") pod \"ce5a9bbf-1908-4220-b871-452bac6bebb8\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " Nov 02 01:54:04.362620 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.362546 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-workload-certs\") pod \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " Nov 02 01:54:04.362620 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.362599 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istiod-ca-cert\") pod \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " Nov 02 01:54:04.362717 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.362629 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce5a9bbf-1908-4220-b871-452bac6bebb8-kube-api-access-pz7l2\" (UniqueName: \"kubernetes.io/projected/ce5a9bbf-1908-4220-b871-452bac6bebb8-kube-api-access-pz7l2\") pod \"ce5a9bbf-1908-4220-b871-452bac6bebb8\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " Nov 02 01:54:04.362717 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.362662 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-podinfo\") pod \"2d7365ed-68db-449b-9ef1-047cc14f55c3\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " Nov 02 01:54:04.362798 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.362741 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce5a9bbf-1908-4220-b871-452bac6bebb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce5a9bbf-1908-4220-b871-452bac6bebb8-istiod-ca-cert\") pod \"ce5a9bbf-1908-4220-b871-452bac6bebb8\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " Nov 02 01:54:04.362798 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.362769 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d7365ed-68db-449b-9ef1-047cc14f55c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7365ed-68db-449b-9ef1-047cc14f55c3-istiod-ca-cert\") pod \"2d7365ed-68db-449b-9ef1-047cc14f55c3\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " Nov 02 01:54:04.362863 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.362849 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-podinfo\") pod \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " Nov 02 01:54:04.362902 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.362877 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7365ed-68db-449b-9ef1-047cc14f55c3-kube-api-access-bztsj\" (UniqueName: \"kubernetes.io/projected/2d7365ed-68db-449b-9ef1-047cc14f55c3-kube-api-access-bztsj\") pod \"2d7365ed-68db-449b-9ef1-047cc14f55c3\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " Nov 02 01:54:04.362902 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.362875 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-workload-certs" pod "ce5a9bbf-1908-4220-b871-452bac6bebb8" (UID: "ce5a9bbf-1908-4220-b871-452bac6bebb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:04.362974 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.362920 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-workload-socket\") pod \"ce5a9bbf-1908-4220-b871-452bac6bebb8\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " Nov 02 01:54:04.362974 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.362966 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-data\") pod \"2d7365ed-68db-449b-9ef1-047cc14f55c3\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " Nov 02 01:54:04.363045 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.362999 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-envoy\") pod \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " Nov 02 01:54:04.363045 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.363029 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-workload-socket\") pod \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " Nov 02 01:54:04.363117 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.363104 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-envoy\") pod \"2d7365ed-68db-449b-9ef1-047cc14f55c3\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " Nov 02 01:54:04.363155 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.363135 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-envoy\") pod \"ce5a9bbf-1908-4220-b871-452bac6bebb8\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " Nov 02 01:54:04.363155 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.363137 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-workload-socket" pod "2d7365ed-68db-449b-9ef1-047cc14f55c3" (UID: "2d7365ed-68db-449b-9ef1-047cc14f55c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:04.363227 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.363194 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-credential-socket\") pod \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " Nov 02 01:54:04.363263 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.363254 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-token\" (UniqueName: \"kubernetes.io/projected/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-token\") pod \"ce5a9bbf-1908-4220-b871-452bac6bebb8\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " Nov 02 01:54:04.363348 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.363283 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-workload-certs\") pod \"2d7365ed-68db-449b-9ef1-047cc14f55c3\" (UID: \"2d7365ed-68db-449b-9ef1-047cc14f55c3\") " Nov 02 01:54:04.363348 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.363325 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-kube-api-access-wggb8\" (UniqueName: \"kubernetes.io/projected/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-kube-api-access-wggb8\") pod \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\" (UID: \"1dbf2ea9-cb23-4fa2-b246-58ca09066b74\") " Nov 02 01:54:04.363416 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.363381 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-podinfo\") pod \"ce5a9bbf-1908-4220-b871-452bac6bebb8\" (UID: \"ce5a9bbf-1908-4220-b871-452bac6bebb8\") " Nov 02 01:54:04.363416 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.363404 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-credential-socket" pod "2d7365ed-68db-449b-9ef1-047cc14f55c3" (UID: "2d7365ed-68db-449b-9ef1-047cc14f55c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:04.364180 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.364156 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.364358 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.364183 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.364358 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.364199 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.365769 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.365502 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-credential-socket" pod "ce5a9bbf-1908-4220-b871-452bac6bebb8" (UID: "ce5a9bbf-1908-4220-b871-452bac6bebb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:04.366230 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.365938 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-data" pod "2d7365ed-68db-449b-9ef1-047cc14f55c3" (UID: "2d7365ed-68db-449b-9ef1-047cc14f55c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:04.366608 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.366377 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-workload-certs" pod "1dbf2ea9-cb23-4fa2-b246-58ca09066b74" (UID: "1dbf2ea9-cb23-4fa2-b246-58ca09066b74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:04.366923 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.366766 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-data" pod "1dbf2ea9-cb23-4fa2-b246-58ca09066b74" (UID: "1dbf2ea9-cb23-4fa2-b246-58ca09066b74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:04.367276 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.367032 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-data" pod "ce5a9bbf-1908-4220-b871-452bac6bebb8" (UID: "ce5a9bbf-1908-4220-b871-452bac6bebb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:04.367276 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.367092 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-envoy" pod "2d7365ed-68db-449b-9ef1-047cc14f55c3" (UID: "2d7365ed-68db-449b-9ef1-047cc14f55c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:04.368526 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.368506 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-credential-socket" pod "1dbf2ea9-cb23-4fa2-b246-58ca09066b74" (UID: "1dbf2ea9-cb23-4fa2-b246-58ca09066b74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:04.368714 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.368681 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-workload-socket" pod "1dbf2ea9-cb23-4fa2-b246-58ca09066b74" (UID: "1dbf2ea9-cb23-4fa2-b246-58ca09066b74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:04.369230 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.369172 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istiod-ca-cert" pod "1dbf2ea9-cb23-4fa2-b246-58ca09066b74" (UID: "1dbf2ea9-cb23-4fa2-b246-58ca09066b74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:54:04.369387 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.369363 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-workload-socket" pod "ce5a9bbf-1908-4220-b871-452bac6bebb8" (UID: "ce5a9bbf-1908-4220-b871-452bac6bebb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:04.369725 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.369694 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-envoy" pod "ce5a9bbf-1908-4220-b871-452bac6bebb8" (UID: "ce5a9bbf-1908-4220-b871-452bac6bebb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:04.370085 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.370057 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-token" pod "1dbf2ea9-cb23-4fa2-b246-58ca09066b74" (UID: "1dbf2ea9-cb23-4fa2-b246-58ca09066b74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:54:04.370736 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.370413 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7365ed-68db-449b-9ef1-047cc14f55c3-istiod-ca-cert" pod "2d7365ed-68db-449b-9ef1-047cc14f55c3" (UID: "2d7365ed-68db-449b-9ef1-047cc14f55c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:54:04.370736 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.370524 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-workload-certs" pod "2d7365ed-68db-449b-9ef1-047cc14f55c3" (UID: "2d7365ed-68db-449b-9ef1-047cc14f55c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:04.370913 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.370745 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce5a9bbf-1908-4220-b871-452bac6bebb8-istiod-ca-cert" pod "ce5a9bbf-1908-4220-b871-452bac6bebb8" (UID: "ce5a9bbf-1908-4220-b871-452bac6bebb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:54:04.372764 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.371825 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-envoy" pod "1dbf2ea9-cb23-4fa2-b246-58ca09066b74" (UID: "1dbf2ea9-cb23-4fa2-b246-58ca09066b74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:04.372764 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.372641 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-podinfo" pod "1dbf2ea9-cb23-4fa2-b246-58ca09066b74" (UID: "1dbf2ea9-cb23-4fa2-b246-58ca09066b74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:54:04.373033 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.372743 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-token" pod "2d7365ed-68db-449b-9ef1-047cc14f55c3" (UID: "2d7365ed-68db-449b-9ef1-047cc14f55c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:54:04.373154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.373112 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-token" pod "ce5a9bbf-1908-4220-b871-452bac6bebb8" (UID: "ce5a9bbf-1908-4220-b871-452bac6bebb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:54:04.373372 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.373347 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-podinfo" pod "ce5a9bbf-1908-4220-b871-452bac6bebb8" (UID: "ce5a9bbf-1908-4220-b871-452bac6bebb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:54:04.373812 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.373649 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-podinfo" pod "2d7365ed-68db-449b-9ef1-047cc14f55c3" (UID: "2d7365ed-68db-449b-9ef1-047cc14f55c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:54:04.373971 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.373927 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7365ed-68db-449b-9ef1-047cc14f55c3-kube-api-access-bztsj" pod "2d7365ed-68db-449b-9ef1-047cc14f55c3" (UID: "2d7365ed-68db-449b-9ef1-047cc14f55c3"). InnerVolumeSpecName "kube-api-access-bztsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:54:04.375006 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.374977 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce5a9bbf-1908-4220-b871-452bac6bebb8-kube-api-access-pz7l2" pod "ce5a9bbf-1908-4220-b871-452bac6bebb8" (UID: "ce5a9bbf-1908-4220-b871-452bac6bebb8"). InnerVolumeSpecName "kube-api-access-pz7l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:54:04.375145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.375126 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-kube-api-access-wggb8" pod "1dbf2ea9-cb23-4fa2-b246-58ca09066b74" (UID: "1dbf2ea9-cb23-4fa2-b246-58ca09066b74"). InnerVolumeSpecName "kube-api-access-wggb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:54:04.384041 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.383910 5044 generic.go:356] "Generic (PLEG): container finished" podID="e541a05c-0d9f-420f-af63-638e2642c6aa" containerID="04d69b4c5a6c8affc1a365226c709c8731d992c16f31e32bc2de72122fdf62bb" exitCode=0 Nov 02 01:54:04.384041 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.383968 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" Nov 02 01:54:04.384041 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.383992 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" event={"ID":"e541a05c-0d9f-420f-af63-638e2642c6aa","Type":"ContainerDied","Data":"04d69b4c5a6c8affc1a365226c709c8731d992c16f31e32bc2de72122fdf62bb"} Nov 02 01:54:04.384264 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.384052 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg" event={"ID":"e541a05c-0d9f-420f-af63-638e2642c6aa","Type":"ContainerDied","Data":"5f22453872807d7e1f711a2aba3fc9a0520daa1f920f4ad118826cc406635df2"} Nov 02 01:54:04.384264 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.384084 5044 scope.go:122] "RemoveContainer" containerID="04d69b4c5a6c8affc1a365226c709c8731d992c16f31e32bc2de72122fdf62bb" Nov 02 01:54:04.392525 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.387574 5044 generic.go:356] "Generic (PLEG): container finished" podID="1dbf2ea9-cb23-4fa2-b246-58ca09066b74" containerID="d907db85be7e47d9345985af951de423ee97fdec394c1640192ca41bdaffa41f" exitCode=0 Nov 02 01:54:04.392525 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.387706 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" event={"ID":"1dbf2ea9-cb23-4fa2-b246-58ca09066b74","Type":"ContainerDied","Data":"d907db85be7e47d9345985af951de423ee97fdec394c1640192ca41bdaffa41f"} Nov 02 01:54:04.392525 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.387745 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" event={"ID":"1dbf2ea9-cb23-4fa2-b246-58ca09066b74","Type":"ContainerDied","Data":"9bfbbea2ae9dc49c2a85d3a5b978f56dcb0ba28052abc5d6b4b93c0d84266987"} Nov 02 01:54:04.392525 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.387819 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k" Nov 02 01:54:04.401838 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.401105 5044 generic.go:356] "Generic (PLEG): container finished" podID="2d7365ed-68db-449b-9ef1-047cc14f55c3" containerID="86412078dfd576c857e7ae73bc01d9684ce0c333c38094343c81e3ce1ea6ea23" exitCode=0 Nov 02 01:54:04.401838 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.401213 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" event={"ID":"2d7365ed-68db-449b-9ef1-047cc14f55c3","Type":"ContainerDied","Data":"86412078dfd576c857e7ae73bc01d9684ce0c333c38094343c81e3ce1ea6ea23"} Nov 02 01:54:04.401838 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.401252 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" event={"ID":"2d7365ed-68db-449b-9ef1-047cc14f55c3","Type":"ContainerDied","Data":"e71951af94d091c41f4502789bcaca7f05984135cabfc071b42892894103ed0c"} Nov 02 01:54:04.401838 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.401331 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284" Nov 02 01:54:04.412593 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.408205 5044 generic.go:356] "Generic (PLEG): container finished" podID="ce5a9bbf-1908-4220-b871-452bac6bebb8" containerID="de1c9b12ae2050bdc1a6d7db5c9da8477ae71fd00d506495d5c1e7885f40cc1e" exitCode=0 Nov 02 01:54:04.412593 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.408250 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" Nov 02 01:54:04.412593 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.408257 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" event={"ID":"ce5a9bbf-1908-4220-b871-452bac6bebb8","Type":"ContainerDied","Data":"de1c9b12ae2050bdc1a6d7db5c9da8477ae71fd00d506495d5c1e7885f40cc1e"} Nov 02 01:54:04.412593 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.408294 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp" event={"ID":"ce5a9bbf-1908-4220-b871-452bac6bebb8","Type":"ContainerDied","Data":"29dcdd94198eeab0dfdda4d2405bfb355a87b578d55e0fd377132db3bb8c2c09"} Nov 02 01:54:04.412593 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.409154 5044 scope.go:122] "RemoveContainer" containerID="04d69b4c5a6c8affc1a365226c709c8731d992c16f31e32bc2de72122fdf62bb" Nov 02 01:54:04.413028 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:54:04.413001 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d69b4c5a6c8affc1a365226c709c8731d992c16f31e32bc2de72122fdf62bb\": container with ID starting with 04d69b4c5a6c8affc1a365226c709c8731d992c16f31e32bc2de72122fdf62bb not found: ID does not exist" containerID="04d69b4c5a6c8affc1a365226c709c8731d992c16f31e32bc2de72122fdf62bb" Nov 02 01:54:04.413075 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.413038 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d69b4c5a6c8affc1a365226c709c8731d992c16f31e32bc2de72122fdf62bb"} err="failed to get container status \"04d69b4c5a6c8affc1a365226c709c8731d992c16f31e32bc2de72122fdf62bb\": rpc error: code = NotFound desc = could not find container \"04d69b4c5a6c8affc1a365226c709c8731d992c16f31e32bc2de72122fdf62bb\": container with ID starting with 04d69b4c5a6c8affc1a365226c709c8731d992c16f31e32bc2de72122fdf62bb not found: ID does not exist" Nov 02 01:54:04.413075 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.413060 5044 scope.go:122] "RemoveContainer" containerID="d907db85be7e47d9345985af951de423ee97fdec394c1640192ca41bdaffa41f" Nov 02 01:54:04.424877 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.424856 5044 scope.go:122] "RemoveContainer" containerID="d907db85be7e47d9345985af951de423ee97fdec394c1640192ca41bdaffa41f" Nov 02 01:54:04.425118 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:54:04.425102 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d907db85be7e47d9345985af951de423ee97fdec394c1640192ca41bdaffa41f\": container with ID starting with d907db85be7e47d9345985af951de423ee97fdec394c1640192ca41bdaffa41f not found: ID does not exist" containerID="d907db85be7e47d9345985af951de423ee97fdec394c1640192ca41bdaffa41f" Nov 02 01:54:04.425161 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.425124 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d907db85be7e47d9345985af951de423ee97fdec394c1640192ca41bdaffa41f"} err="failed to get container status \"d907db85be7e47d9345985af951de423ee97fdec394c1640192ca41bdaffa41f\": rpc error: code = NotFound desc = could not find container \"d907db85be7e47d9345985af951de423ee97fdec394c1640192ca41bdaffa41f\": container with ID starting with d907db85be7e47d9345985af951de423ee97fdec394c1640192ca41bdaffa41f not found: ID does not exist" Nov 02 01:54:04.425161 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.425138 5044 scope.go:122] "RemoveContainer" containerID="86412078dfd576c857e7ae73bc01d9684ce0c333c38094343c81e3ce1ea6ea23" Nov 02 01:54:04.448752 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.448726 5044 scope.go:122] "RemoveContainer" containerID="86412078dfd576c857e7ae73bc01d9684ce0c333c38094343c81e3ce1ea6ea23" Nov 02 01:54:04.449101 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:54:04.449081 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86412078dfd576c857e7ae73bc01d9684ce0c333c38094343c81e3ce1ea6ea23\": container with ID starting with 86412078dfd576c857e7ae73bc01d9684ce0c333c38094343c81e3ce1ea6ea23 not found: ID does not exist" containerID="86412078dfd576c857e7ae73bc01d9684ce0c333c38094343c81e3ce1ea6ea23" Nov 02 01:54:04.449148 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.449109 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86412078dfd576c857e7ae73bc01d9684ce0c333c38094343c81e3ce1ea6ea23"} err="failed to get container status \"86412078dfd576c857e7ae73bc01d9684ce0c333c38094343c81e3ce1ea6ea23\": rpc error: code = NotFound desc = could not find container \"86412078dfd576c857e7ae73bc01d9684ce0c333c38094343c81e3ce1ea6ea23\": container with ID starting with 86412078dfd576c857e7ae73bc01d9684ce0c333c38094343c81e3ce1ea6ea23 not found: ID does not exist" Nov 02 01:54:04.449148 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.449127 5044 scope.go:122] "RemoveContainer" containerID="de1c9b12ae2050bdc1a6d7db5c9da8477ae71fd00d506495d5c1e7885f40cc1e" Nov 02 01:54:04.466029 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.464765 5044 scope.go:122] "RemoveContainer" containerID="de1c9b12ae2050bdc1a6d7db5c9da8477ae71fd00d506495d5c1e7885f40cc1e" Nov 02 01:54:04.468023 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:54:04.467462 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1c9b12ae2050bdc1a6d7db5c9da8477ae71fd00d506495d5c1e7885f40cc1e\": container with ID starting with de1c9b12ae2050bdc1a6d7db5c9da8477ae71fd00d506495d5c1e7885f40cc1e not found: ID does not exist" containerID="de1c9b12ae2050bdc1a6d7db5c9da8477ae71fd00d506495d5c1e7885f40cc1e" Nov 02 01:54:04.468023 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.467526 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1c9b12ae2050bdc1a6d7db5c9da8477ae71fd00d506495d5c1e7885f40cc1e"} err="failed to get container status \"de1c9b12ae2050bdc1a6d7db5c9da8477ae71fd00d506495d5c1e7885f40cc1e\": rpc error: code = NotFound desc = could not find container \"de1c9b12ae2050bdc1a6d7db5c9da8477ae71fd00d506495d5c1e7885f40cc1e\": container with ID starting with de1c9b12ae2050bdc1a6d7db5c9da8477ae71fd00d506495d5c1e7885f40cc1e not found: ID does not exist" Nov 02 01:54:04.469607 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.469588 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.469781 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.469766 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.469887 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.469875 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wggb8\" (UniqueName: \"kubernetes.io/projected/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-kube-api-access-wggb8\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.469963 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.469953 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.470038 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.470028 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.470112 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.470100 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.470189 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.470179 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.470270 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.470259 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.470349 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.470339 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.470424 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.470412 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.470500 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.470489 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.470572 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.470562 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz7l2\" (UniqueName: \"kubernetes.io/projected/ce5a9bbf-1908-4220-b871-452bac6bebb8-kube-api-access-pz7l2\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.470649 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.470639 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.470749 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.470737 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce5a9bbf-1908-4220-b871-452bac6bebb8-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.470835 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.470824 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d7365ed-68db-449b-9ef1-047cc14f55c3-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.470922 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.470907 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.471008 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.470995 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bztsj\" (UniqueName: \"kubernetes.io/projected/2d7365ed-68db-449b-9ef1-047cc14f55c3-kube-api-access-bztsj\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.471084 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.471074 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.471167 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.471157 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.471250 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.471239 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.471334 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.471323 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.471422 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.471411 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d7365ed-68db-449b-9ef1-047cc14f55c3-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.471499 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.471488 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce5a9bbf-1908-4220-b871-452bac6bebb8-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.471582 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.471570 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbf2ea9-cb23-4fa2-b246-58ca09066b74-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:04.760497 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.760468 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284"] Nov 02 01:54:04.771231 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.771202 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hm284"] Nov 02 01:54:04.883299 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.883264 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg"] Nov 02 01:54:04.888092 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.888062 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-hd5dg"] Nov 02 01:54:04.996765 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:04.996729 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp"] Nov 02 01:54:05.000113 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:05.000087 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-qs2bp"] Nov 02 01:54:05.113091 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:05.113010 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k"] Nov 02 01:54:05.116526 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:05.116497 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-88e76d-openshift-default-66dc56b97f-drf8k"] Nov 02 01:54:05.455057 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:05.454976 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dbf2ea9-cb23-4fa2-b246-58ca09066b74" path="/var/lib/kubelet/pods/1dbf2ea9-cb23-4fa2-b246-58ca09066b74/volumes" Nov 02 01:54:05.455494 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:05.455480 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d7365ed-68db-449b-9ef1-047cc14f55c3" path="/var/lib/kubelet/pods/2d7365ed-68db-449b-9ef1-047cc14f55c3/volumes" Nov 02 01:54:05.455989 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:05.455974 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce5a9bbf-1908-4220-b871-452bac6bebb8" path="/var/lib/kubelet/pods/ce5a9bbf-1908-4220-b871-452bac6bebb8/volumes" Nov 02 01:54:05.456782 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:05.456766 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e541a05c-0d9f-420f-af63-638e2642c6aa" path="/var/lib/kubelet/pods/e541a05c-0d9f-420f-af63-638e2642c6aa/volumes" Nov 02 01:54:05.750748 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:05.749701 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:06.707425 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:06.707387 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:11.941811 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:11.941759 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:12.344390 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.344360 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct"] Nov 02 01:54:12.345069 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.345053 5044 cpu_manager.go:414] "containerMap: removing container" podUID="2d7365ed-68db-449b-9ef1-047cc14f55c3" containerName="istio-proxy" Nov 02 01:54:12.345133 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.345070 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7365ed-68db-449b-9ef1-047cc14f55c3" containerName="istio-proxy" Nov 02 01:54:12.345133 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.345088 5044 cpu_manager.go:414] "containerMap: removing container" podUID="1dbf2ea9-cb23-4fa2-b246-58ca09066b74" containerName="istio-proxy" Nov 02 01:54:12.345133 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.345094 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dbf2ea9-cb23-4fa2-b246-58ca09066b74" containerName="istio-proxy" Nov 02 01:54:12.345133 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.345111 5044 cpu_manager.go:414] "containerMap: removing container" podUID="e541a05c-0d9f-420f-af63-638e2642c6aa" containerName="istio-proxy" Nov 02 01:54:12.345133 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.345117 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e541a05c-0d9f-420f-af63-638e2642c6aa" containerName="istio-proxy" Nov 02 01:54:12.345133 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.345134 5044 cpu_manager.go:414] "containerMap: removing container" podUID="ce5a9bbf-1908-4220-b871-452bac6bebb8" containerName="istio-proxy" Nov 02 01:54:12.345322 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.345139 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce5a9bbf-1908-4220-b871-452bac6bebb8" containerName="istio-proxy" Nov 02 01:54:12.345322 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.345232 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce5a9bbf-1908-4220-b871-452bac6bebb8" containerName="istio-proxy" Nov 02 01:54:12.345322 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.345246 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7365ed-68db-449b-9ef1-047cc14f55c3" containerName="istio-proxy" Nov 02 01:54:12.345322 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.345254 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dbf2ea9-cb23-4fa2-b246-58ca09066b74" containerName="istio-proxy" Nov 02 01:54:12.345322 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.345262 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="e541a05c-0d9f-420f-af63-638e2642c6aa" containerName="istio-proxy" Nov 02 01:54:12.350456 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.350436 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" Nov 02 01:54:12.407333 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.407309 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct"] Nov 02 01:54:12.453432 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.453403 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-d700e9-serving-cert\"" Nov 02 01:54:12.484717 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.477868 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83df443-031e-4a2e-a4f4-5a686ad12085-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct\" (UID: \"a83df443-031e-4a2e-a4f4-5a686ad12085\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" Nov 02 01:54:12.484717 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.478166 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hv87\" (UniqueName: \"kubernetes.io/projected/a83df443-031e-4a2e-a4f4-5a686ad12085-kube-api-access-5hv87\") pod \"maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct\" (UID: \"a83df443-031e-4a2e-a4f4-5a686ad12085\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" Nov 02 01:54:12.484717 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.478216 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83df443-031e-4a2e-a4f4-5a686ad12085-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct\" (UID: \"a83df443-031e-4a2e-a4f4-5a686ad12085\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" Nov 02 01:54:12.484717 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.478287 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83df443-031e-4a2e-a4f4-5a686ad12085-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct\" (UID: \"a83df443-031e-4a2e-a4f4-5a686ad12085\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" Nov 02 01:54:12.540270 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.540241 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:12.580340 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.580307 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hv87\" (UniqueName: \"kubernetes.io/projected/a83df443-031e-4a2e-a4f4-5a686ad12085-kube-api-access-5hv87\") pod \"maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct\" (UID: \"a83df443-031e-4a2e-a4f4-5a686ad12085\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" Nov 02 01:54:12.580488 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.580358 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83df443-031e-4a2e-a4f4-5a686ad12085-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct\" (UID: \"a83df443-031e-4a2e-a4f4-5a686ad12085\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" Nov 02 01:54:12.580718 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.580695 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83df443-031e-4a2e-a4f4-5a686ad12085-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct\" (UID: \"a83df443-031e-4a2e-a4f4-5a686ad12085\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" Nov 02 01:54:12.580986 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.580955 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83df443-031e-4a2e-a4f4-5a686ad12085-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct\" (UID: \"a83df443-031e-4a2e-a4f4-5a686ad12085\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" Nov 02 01:54:12.581339 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:54:12.581319 5044 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-serving-cert: secret "maas-api-e2e-shared-a-w4-d700e9-serving-cert" not found Nov 02 01:54:12.581433 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:54:12.581419 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83df443-031e-4a2e-a4f4-5a686ad12085-maas-api-tls podName:a83df443-031e-4a2e-a4f4-5a686ad12085 nodeName:}" failed. No retries permitted until 2026-11-02 01:54:13.081397484 +0000 UTC m=+1769.752650157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a83df443-031e-4a2e-a4f4-5a686ad12085-maas-api-tls") pod "maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" (UID: "a83df443-031e-4a2e-a4f4-5a686ad12085") : secret "maas-api-e2e-shared-a-w4-d700e9-serving-cert" not found Nov 02 01:54:12.582180 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.582154 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83df443-031e-4a2e-a4f4-5a686ad12085-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct\" (UID: \"a83df443-031e-4a2e-a4f4-5a686ad12085\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" Nov 02 01:54:12.583222 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.583202 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83df443-031e-4a2e-a4f4-5a686ad12085-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct\" (UID: \"a83df443-031e-4a2e-a4f4-5a686ad12085\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" Nov 02 01:54:12.644840 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.644471 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hv87\" (UniqueName: \"kubernetes.io/projected/a83df443-031e-4a2e-a4f4-5a686ad12085-kube-api-access-5hv87\") pod \"maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct\" (UID: \"a83df443-031e-4a2e-a4f4-5a686ad12085\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" Nov 02 01:54:12.702127 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.702096 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:12.850167 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:12.850134 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:13.091183 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:13.091150 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83df443-031e-4a2e-a4f4-5a686ad12085-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct\" (UID: \"a83df443-031e-4a2e-a4f4-5a686ad12085\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" Nov 02 01:54:13.104718 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:13.102240 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83df443-031e-4a2e-a4f4-5a686ad12085-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct\" (UID: \"a83df443-031e-4a2e-a4f4-5a686ad12085\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" Nov 02 01:54:13.264451 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:13.264414 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" Nov 02 01:54:13.681834 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:13.681804 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct"] Nov 02 01:54:13.682169 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:54:13.682132 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83df443_031e_4a2e_a4f4_5a686ad12085.slice/crio-936db0206058a44babe29a8dcf574ec8ce8971dddb251051e2f730dc8430b248 WatchSource:0}: Error finding container 936db0206058a44babe29a8dcf574ec8ce8971dddb251051e2f730dc8430b248: Status 404 returned error can't find the container with id 936db0206058a44babe29a8dcf574ec8ce8971dddb251051e2f730dc8430b248 Nov 02 01:54:14.531384 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:14.531345 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" event={"ID":"a83df443-031e-4a2e-a4f4-5a686ad12085","Type":"ContainerStarted","Data":"936db0206058a44babe29a8dcf574ec8ce8971dddb251051e2f730dc8430b248"} Nov 02 01:54:15.541560 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:15.541512 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" event={"ID":"a83df443-031e-4a2e-a4f4-5a686ad12085","Type":"ContainerStarted","Data":"1826b0770551eaf7962311e087b7947c340cb1a7b0d69a561c88bffab28f801c"} Nov 02 01:54:15.542038 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:15.541584 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" Nov 02 01:54:15.664602 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:15.664549 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" podStartSLOduration=2.577997334 podStartE2EDuration="3.664534725s" totalImagesPullingTime="1.086537391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:15.660708554 +0000 UTC m=+1772.331961234" watchObservedRunningTime="2026-11-02 01:54:15.664534725 +0000 UTC m=+1772.335787405" Nov 02 01:54:16.938179 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:16.938143 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6"] Nov 02 01:54:16.938704 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:16.938394 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" podUID="4ac69c4e-27a7-41b7-a2d5-dfce81e50934" containerName="maas-api" containerID="cri-o://591b213b450e4a7b51a96f5b8a7bf0cc38b82be0b31f80cb5c8e3435868bf1b9" gracePeriod=30 Nov 02 01:54:17.064519 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.064421 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:17.316868 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.316839 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" Nov 02 01:54:17.384409 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.384370 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-metrics-certs\") pod \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\" (UID: \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\") " Nov 02 01:54:17.384923 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.384898 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-openshift-service-ca\") pod \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\" (UID: \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\") " Nov 02 01:54:17.385073 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.385059 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-maas-api-tls\") pod \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\" (UID: \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\") " Nov 02 01:54:17.385445 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.385429 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-kube-api-access-fg7fs\" (UniqueName: \"kubernetes.io/projected/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-kube-api-access-fg7fs\") pod \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\" (UID: \"4ac69c4e-27a7-41b7-a2d5-dfce81e50934\") " Nov 02 01:54:17.385992 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.385589 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-openshift-service-ca" pod "4ac69c4e-27a7-41b7-a2d5-dfce81e50934" (UID: "4ac69c4e-27a7-41b7-a2d5-dfce81e50934"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:54:17.387267 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.387230 5044 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-openshift-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:17.389662 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.389632 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-metrics-certs" pod "4ac69c4e-27a7-41b7-a2d5-dfce81e50934" (UID: "4ac69c4e-27a7-41b7-a2d5-dfce81e50934"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:54:17.390190 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.390151 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-kube-api-access-fg7fs" pod "4ac69c4e-27a7-41b7-a2d5-dfce81e50934" (UID: "4ac69c4e-27a7-41b7-a2d5-dfce81e50934"). InnerVolumeSpecName "kube-api-access-fg7fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:54:17.392841 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.392817 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-maas-api-tls" pod "4ac69c4e-27a7-41b7-a2d5-dfce81e50934" (UID: "4ac69c4e-27a7-41b7-a2d5-dfce81e50934"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 01:54:17.489390 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.489359 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg7fs\" (UniqueName: \"kubernetes.io/projected/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-kube-api-access-fg7fs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:17.489598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.489571 5044 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-metrics-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:17.489720 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.489600 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ac69c4e-27a7-41b7-a2d5-dfce81e50934-maas-api-tls\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:17.559267 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.559179 5044 generic.go:356] "Generic (PLEG): container finished" podID="4ac69c4e-27a7-41b7-a2d5-dfce81e50934" containerID="591b213b450e4a7b51a96f5b8a7bf0cc38b82be0b31f80cb5c8e3435868bf1b9" exitCode=0 Nov 02 01:54:17.559267 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.559223 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" Nov 02 01:54:17.559483 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.559259 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" event={"ID":"4ac69c4e-27a7-41b7-a2d5-dfce81e50934","Type":"ContainerDied","Data":"591b213b450e4a7b51a96f5b8a7bf0cc38b82be0b31f80cb5c8e3435868bf1b9"} Nov 02 01:54:17.559483 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.559320 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6" event={"ID":"4ac69c4e-27a7-41b7-a2d5-dfce81e50934","Type":"ContainerDied","Data":"d4584bd1fb7ae734b892f9f9ff1845968adac9af84a0df9c9c5faa0faec3d2a1"} Nov 02 01:54:17.559483 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.559349 5044 scope.go:122] "RemoveContainer" containerID="591b213b450e4a7b51a96f5b8a7bf0cc38b82be0b31f80cb5c8e3435868bf1b9" Nov 02 01:54:17.571102 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.571084 5044 scope.go:122] "RemoveContainer" containerID="591b213b450e4a7b51a96f5b8a7bf0cc38b82be0b31f80cb5c8e3435868bf1b9" Nov 02 01:54:17.571399 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:54:17.571380 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"591b213b450e4a7b51a96f5b8a7bf0cc38b82be0b31f80cb5c8e3435868bf1b9\": container with ID starting with 591b213b450e4a7b51a96f5b8a7bf0cc38b82be0b31f80cb5c8e3435868bf1b9 not found: ID does not exist" containerID="591b213b450e4a7b51a96f5b8a7bf0cc38b82be0b31f80cb5c8e3435868bf1b9" Nov 02 01:54:17.571467 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.571409 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"591b213b450e4a7b51a96f5b8a7bf0cc38b82be0b31f80cb5c8e3435868bf1b9"} err="failed to get container status \"591b213b450e4a7b51a96f5b8a7bf0cc38b82be0b31f80cb5c8e3435868bf1b9\": rpc error: code = NotFound desc = could not find container \"591b213b450e4a7b51a96f5b8a7bf0cc38b82be0b31f80cb5c8e3435868bf1b9\": container with ID starting with 591b213b450e4a7b51a96f5b8a7bf0cc38b82be0b31f80cb5c8e3435868bf1b9 not found: ID does not exist" Nov 02 01:54:17.735198 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.735162 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6"] Nov 02 01:54:17.741064 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:17.741035 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8ebd9167-5f9f69655d-s4vn6"] Nov 02 01:54:18.259864 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:18.259827 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:18.439032 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:18.434085 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:18.816453 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:18.815793 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:18.838691 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:18.836286 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg"] Nov 02 01:54:18.838691 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:18.837481 5044 cpu_manager.go:414] "containerMap: removing container" podUID="4ac69c4e-27a7-41b7-a2d5-dfce81e50934" containerName="maas-api" Nov 02 01:54:18.838691 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:18.837502 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac69c4e-27a7-41b7-a2d5-dfce81e50934" containerName="maas-api" Nov 02 01:54:18.838691 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:18.838179 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac69c4e-27a7-41b7-a2d5-dfce81e50934" containerName="maas-api" Nov 02 01:54:18.846133 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:18.846009 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg" Nov 02 01:54:18.903273 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:18.902334 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg"] Nov 02 01:54:18.956698 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:18.944637 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4krj\" (UniqueName: \"kubernetes.io/projected/fcdbd493-2074-4976-aef4-9eb91f6f7522-kube-api-access-l4krj\") pod \"payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg\" (UID: \"fcdbd493-2074-4976-aef4-9eb91f6f7522\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg" Nov 02 01:54:18.956698 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:18.944775 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcdbd493-2074-4976-aef4-9eb91f6f7522-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg\" (UID: \"fcdbd493-2074-4976-aef4-9eb91f6f7522\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg" Nov 02 01:54:18.956698 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:18.945491 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-d700e9-dockercfg-94nng\"" Nov 02 01:54:18.956698 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:18.945560 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-d700e9\"" Nov 02 01:54:19.013924 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.007007 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff"] Nov 02 01:54:19.017050 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.017020 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff" Nov 02 01:54:19.046531 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.046493 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4krj\" (UniqueName: \"kubernetes.io/projected/fcdbd493-2074-4976-aef4-9eb91f6f7522-kube-api-access-l4krj\") pod \"payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg\" (UID: \"fcdbd493-2074-4976-aef4-9eb91f6f7522\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg" Nov 02 01:54:19.046656 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.046603 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcdbd493-2074-4976-aef4-9eb91f6f7522-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg\" (UID: \"fcdbd493-2074-4976-aef4-9eb91f6f7522\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg" Nov 02 01:54:19.047343 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.047323 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcdbd493-2074-4976-aef4-9eb91f6f7522-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg\" (UID: \"fcdbd493-2074-4976-aef4-9eb91f6f7522\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg" Nov 02 01:54:19.071108 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.068304 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff"] Nov 02 01:54:19.090442 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.090413 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:19.109838 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.109638 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4krj\" (UniqueName: \"kubernetes.io/projected/fcdbd493-2074-4976-aef4-9eb91f6f7522-kube-api-access-l4krj\") pod \"payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg\" (UID: \"fcdbd493-2074-4976-aef4-9eb91f6f7522\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg" Nov 02 01:54:19.148327 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.148281 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc7m4\" (UniqueName: \"kubernetes.io/projected/8ee58b67-0d93-4541-b43d-0c8f790ecc38-kube-api-access-sc7m4\") pod \"payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff\" (UID: \"8ee58b67-0d93-4541-b43d-0c8f790ecc38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff" Nov 02 01:54:19.148594 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.148571 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee58b67-0d93-4541-b43d-0c8f790ecc38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff\" (UID: \"8ee58b67-0d93-4541-b43d-0c8f790ecc38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff" Nov 02 01:54:19.171683 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.171635 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg" Nov 02 01:54:19.233871 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.233837 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:19.253778 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.253727 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc7m4\" (UniqueName: \"kubernetes.io/projected/8ee58b67-0d93-4541-b43d-0c8f790ecc38-kube-api-access-sc7m4\") pod \"payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff\" (UID: \"8ee58b67-0d93-4541-b43d-0c8f790ecc38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff" Nov 02 01:54:19.254000 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.253949 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee58b67-0d93-4541-b43d-0c8f790ecc38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff\" (UID: \"8ee58b67-0d93-4541-b43d-0c8f790ecc38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff" Nov 02 01:54:19.254943 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.254907 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee58b67-0d93-4541-b43d-0c8f790ecc38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff\" (UID: \"8ee58b67-0d93-4541-b43d-0c8f790ecc38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff" Nov 02 01:54:19.316836 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.316658 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc7m4\" (UniqueName: \"kubernetes.io/projected/8ee58b67-0d93-4541-b43d-0c8f790ecc38-kube-api-access-sc7m4\") pod \"payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff\" (UID: \"8ee58b67-0d93-4541-b43d-0c8f790ecc38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff" Nov 02 01:54:19.350940 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.338453 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff" Nov 02 01:54:19.462581 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.462551 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:19.473420 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.471926 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ac69c4e-27a7-41b7-a2d5-dfce81e50934" path="/var/lib/kubelet/pods/4ac69c4e-27a7-41b7-a2d5-dfce81e50934/volumes" Nov 02 01:54:19.595187 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.595156 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg"] Nov 02 01:54:19.644978 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.644940 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg" event={"ID":"fcdbd493-2074-4976-aef4-9eb91f6f7522","Type":"ContainerStarted","Data":"3937ab087b8cc5db311da77bf7d47a2a5a3d31c7798f2c375b3e2eae2f727173"} Nov 02 01:54:19.792054 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:19.789938 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff"] Nov 02 01:54:19.796040 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:54:19.796008 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee58b67_0d93_4541_b43d_0c8f790ecc38.slice/crio-077bb70d1b0bb5d0cb9fa04745e7e5366e93f7c74c2010bf1a48ef9b6583109d WatchSource:0}: Error finding container 077bb70d1b0bb5d0cb9fa04745e7e5366e93f7c74c2010bf1a48ef9b6583109d: Status 404 returned error can't find the container with id 077bb70d1b0bb5d0cb9fa04745e7e5366e93f7c74c2010bf1a48ef9b6583109d Nov 02 01:54:20.307834 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:20.307798 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:20.656197 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:20.656097 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg" event={"ID":"fcdbd493-2074-4976-aef4-9eb91f6f7522","Type":"ContainerStarted","Data":"9f64fce5aaead46ed4a390609f7ab42002345f413dfe17bcf2bf60ebae942e21"} Nov 02 01:54:20.656646 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:20.656253 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg" Nov 02 01:54:20.658109 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:20.658081 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff" event={"ID":"8ee58b67-0d93-4541-b43d-0c8f790ecc38","Type":"ContainerStarted","Data":"750282f0f64facdac6be428db50e024cdab4d119601a10f0d75d99958eb9456f"} Nov 02 01:54:20.658234 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:20.658129 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff" event={"ID":"8ee58b67-0d93-4541-b43d-0c8f790ecc38","Type":"ContainerStarted","Data":"077bb70d1b0bb5d0cb9fa04745e7e5366e93f7c74c2010bf1a48ef9b6583109d"} Nov 02 01:54:20.658234 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:20.658162 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff" Nov 02 01:54:20.779111 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:20.779044 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg" podStartSLOduration=2.779025163 podStartE2EDuration="2.779025163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:20.776568108 +0000 UTC m=+1777.447820791" watchObservedRunningTime="2026-11-02 01:54:20.779025163 +0000 UTC m=+1777.450277845" Nov 02 01:54:20.903395 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:20.898638 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff" podStartSLOduration=2.8986205800000002 podStartE2EDuration="2.89862058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:20.895234132 +0000 UTC m=+1777.566486815" watchObservedRunningTime="2026-11-02 01:54:20.89862058 +0000 UTC m=+1777.569873266" Nov 02 01:54:21.032822 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:20.963355 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w"] Nov 02 01:54:21.035814 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.011627 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:21.043770 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.037930 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.121067 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.120900 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-a7c06d-openshift-default-dockercfg-9qncg\"" Nov 02 01:54:21.122486 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.122401 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w"] Nov 02 01:54:21.134926 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.134235 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istiod-ca-cert\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.134926 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.134292 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dlzj\" (UniqueName: \"kubernetes.io/projected/1098cb6c-8b36-4df8-87d9-357d3ed510e0-kube-api-access-7dlzj\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.134926 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.134440 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-credential-socket\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.134926 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.134491 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-data\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.134926 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.134539 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-workload-certs\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.134926 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.134587 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-envoy\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.134926 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.134743 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-workload-socket\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.134926 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.134905 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-podinfo\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.135464 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.134956 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-token\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.236758 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.236727 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istiod-ca-cert\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.236969 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.236762 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dlzj\" (UniqueName: \"kubernetes.io/projected/1098cb6c-8b36-4df8-87d9-357d3ed510e0-kube-api-access-7dlzj\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.236969 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.236837 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-credential-socket\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.237142 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.236973 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-data\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.237142 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.237040 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-workload-certs\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.237142 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.237097 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-envoy\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.237293 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.237146 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-workload-socket\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.237293 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.237226 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-podinfo\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.237293 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.237276 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-token\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.237474 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.237451 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-credential-socket\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.237556 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.237536 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-workload-socket\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.237615 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.237602 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-workload-certs\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.237648 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.237618 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istiod-ca-cert\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.237865 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.237843 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-data\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.239881 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.239854 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-envoy\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.240199 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.240173 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-podinfo\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.318738 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.314309 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dlzj\" (UniqueName: \"kubernetes.io/projected/1098cb6c-8b36-4df8-87d9-357d3ed510e0-kube-api-access-7dlzj\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.319256 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.319205 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-token\") pod \"e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.432094 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.432048 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:21.461543 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.461508 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:21.556060 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.556034 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" Nov 02 01:54:21.898345 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:54:21.898298 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1098cb6c_8b36_4df8_87d9_357d3ed510e0.slice/crio-7dd2705747b176df246fbe0921868bed9cd30258792099a60277fbf8001c2195 WatchSource:0}: Error finding container 7dd2705747b176df246fbe0921868bed9cd30258792099a60277fbf8001c2195: Status 404 returned error can't find the container with id 7dd2705747b176df246fbe0921868bed9cd30258792099a60277fbf8001c2195 Nov 02 01:54:21.898840 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.898626 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w"] Nov 02 01:54:21.902035 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.901998 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:54:21.902170 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.902087 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:54:21.902170 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:21.902136 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:54:22.676002 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:22.675966 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" event={"ID":"1098cb6c-8b36-4df8-87d9-357d3ed510e0","Type":"ContainerStarted","Data":"f9a7355bb7e05b8ad35aaeaf18aab7ec02f1ca3c2d163e0837e5d253a6305359"} Nov 02 01:54:22.676164 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:22.676027 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" event={"ID":"1098cb6c-8b36-4df8-87d9-357d3ed510e0","Type":"ContainerStarted","Data":"7dd2705747b176df246fbe0921868bed9cd30258792099a60277fbf8001c2195"} Nov 02 01:54:22.803024 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:22.802971 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" podStartSLOduration=2.802957062 podStartE2EDuration="2.802957062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:22.79896633 +0000 UTC m=+1779.470219013" watchObservedRunningTime="2026-11-02 01:54:22.802957062 +0000 UTC m=+1779.474209740" Nov 02 01:54:23.433220 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:23.433188 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:23.439725 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:23.439701 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:23.683736 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:23.683640 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:23.684778 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:23.684758 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:54:24.176351 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:54:24.176316 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:54:24.620212 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:24.620178 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:26.387554 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:26.387521 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:26.536233 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:26.536204 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:28.352114 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:28.352077 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657"] Nov 02 01:54:28.352530 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:28.352299 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" podUID="b7d46753-0b7a-4c8c-9870-31cedadbe4c5" containerName="istio-proxy" containerID="cri-o://215d1088a6457e2da21f285503ff09dca44db2d4b7eda1d4c13e2afff4a6915b" gracePeriod=30 Nov 02 01:54:28.405131 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:28.404703 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw"] Nov 02 01:54:28.405131 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:28.404962 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" podUID="25c9e11e-b82c-4e33-81bb-1ccae0588c2d" containerName="istio-proxy" containerID="cri-o://84ca97d6aef5c1718e897f117e436a737109df42e4486966e9497f3dc89d1b46" gracePeriod=30 Nov 02 01:54:28.410610 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:28.410574 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559"] Nov 02 01:54:28.410858 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:28.410834 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" podUID="f9008e53-2fc7-44ab-b841-e4eea224fded" containerName="istio-proxy" containerID="cri-o://89ef6add7905cab77444f3b6a5a42d0e02c466bb33e76e4b7fc9a80138f82426" gracePeriod=30 Nov 02 01:54:28.453311 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:28.453280 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f"] Nov 02 01:54:28.453511 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:28.453473 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" podUID="98be750b-4e74-4378-871b-3a3e9d53cd78" containerName="istio-proxy" containerID="cri-o://f49c280fc18c14ccfa7c4c3c659dabc621db819e2a3200589db9c35475e37122" gracePeriod=30 Nov 02 01:54:29.995355 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:29.995322 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:31.668784 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:31.668752 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff" Nov 02 01:54:31.669213 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:31.668820 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg" Nov 02 01:54:31.996604 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:31.996517 5044 patch_prober.go:29] interesting pod/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:54:31.996604 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:31.996592 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" podUID="98be750b-4e74-4378-871b-3a3e9d53cd78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:54:33.781796 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.781578 5044 generic.go:356] "Generic (PLEG): container finished" podID="f9008e53-2fc7-44ab-b841-e4eea224fded" containerID="89ef6add7905cab77444f3b6a5a42d0e02c466bb33e76e4b7fc9a80138f82426" exitCode=0 Nov 02 01:54:33.781796 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.781730 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" event={"ID":"f9008e53-2fc7-44ab-b841-e4eea224fded","Type":"ContainerDied","Data":"89ef6add7905cab77444f3b6a5a42d0e02c466bb33e76e4b7fc9a80138f82426"} Nov 02 01:54:33.784915 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.784654 5044 generic.go:356] "Generic (PLEG): container finished" podID="98be750b-4e74-4378-871b-3a3e9d53cd78" containerID="f49c280fc18c14ccfa7c4c3c659dabc621db819e2a3200589db9c35475e37122" exitCode=0 Nov 02 01:54:33.784915 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.784832 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" event={"ID":"98be750b-4e74-4378-871b-3a3e9d53cd78","Type":"ContainerDied","Data":"f49c280fc18c14ccfa7c4c3c659dabc621db819e2a3200589db9c35475e37122"} Nov 02 01:54:33.787208 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.787057 5044 generic.go:356] "Generic (PLEG): container finished" podID="25c9e11e-b82c-4e33-81bb-1ccae0588c2d" containerID="84ca97d6aef5c1718e897f117e436a737109df42e4486966e9497f3dc89d1b46" exitCode=0 Nov 02 01:54:33.787208 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.787136 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" event={"ID":"25c9e11e-b82c-4e33-81bb-1ccae0588c2d","Type":"ContainerDied","Data":"84ca97d6aef5c1718e897f117e436a737109df42e4486966e9497f3dc89d1b46"} Nov 02 01:54:33.790435 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.790177 5044 generic.go:356] "Generic (PLEG): container finished" podID="b7d46753-0b7a-4c8c-9870-31cedadbe4c5" containerID="215d1088a6457e2da21f285503ff09dca44db2d4b7eda1d4c13e2afff4a6915b" exitCode=0 Nov 02 01:54:33.790435 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.790253 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" event={"ID":"b7d46753-0b7a-4c8c-9870-31cedadbe4c5","Type":"ContainerDied","Data":"215d1088a6457e2da21f285503ff09dca44db2d4b7eda1d4c13e2afff4a6915b"} Nov 02 01:54:33.790435 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.790288 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" event={"ID":"b7d46753-0b7a-4c8c-9870-31cedadbe4c5","Type":"ContainerDied","Data":"21314af556ab56dcc6833169de06ae0a1489f288701ed48cce35b4deaa7f67d2"} Nov 02 01:54:33.790435 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.790309 5044 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21314af556ab56dcc6833169de06ae0a1489f288701ed48cce35b4deaa7f67d2" Nov 02 01:54:33.790435 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.790324 5044 scope.go:122] "RemoveContainer" containerID="b490bbb8a5b53dfe58e293e00dcb367abc4b391097731dc7baa85a5acc11329b" Nov 02 01:54:33.793733 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.793577 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:54:33.814740 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.812713 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:54:33.873629 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.873600 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:54:33.889929 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.889904 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:54:33.953488 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.953390 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9008e53-2fc7-44ab-b841-e4eea224fded-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9008e53-2fc7-44ab-b841-e4eea224fded-istiod-ca-cert\") pod \"f9008e53-2fc7-44ab-b841-e4eea224fded\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.953500 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-token\" (UniqueName: \"kubernetes.io/projected/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-token\") pod \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.953599 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-data\") pod \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.953629 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-data\") pod \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.953662 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-envoy\") pod \"f9008e53-2fc7-44ab-b841-e4eea224fded\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.953984 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-data" pod "b7d46753-0b7a-4c8c-9870-31cedadbe4c5" (UID: "b7d46753-0b7a-4c8c-9870-31cedadbe4c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954015 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-workload-socket\") pod \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954080 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-credential-socket\") pod \"f9008e53-2fc7-44ab-b841-e4eea224fded\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954161 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-workload-certs\") pod \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954214 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-token\" (UniqueName: \"kubernetes.io/projected/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-token\") pod \"f9008e53-2fc7-44ab-b841-e4eea224fded\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954242 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9008e53-2fc7-44ab-b841-e4eea224fded-kube-api-access-lsk4w\" (UniqueName: \"kubernetes.io/projected/f9008e53-2fc7-44ab-b841-e4eea224fded-kube-api-access-lsk4w\") pod \"f9008e53-2fc7-44ab-b841-e4eea224fded\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954290 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-data" pod "25c9e11e-b82c-4e33-81bb-1ccae0588c2d" (UID: "25c9e11e-b82c-4e33-81bb-1ccae0588c2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954296 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-envoy\") pod \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954292 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-workload-socket" pod "b7d46753-0b7a-4c8c-9870-31cedadbe4c5" (UID: "b7d46753-0b7a-4c8c-9870-31cedadbe4c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954344 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-workload-certs\") pod \"f9008e53-2fc7-44ab-b841-e4eea224fded\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954394 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-workload-socket\") pod \"f9008e53-2fc7-44ab-b841-e4eea224fded\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954411 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-credential-socket" pod "f9008e53-2fc7-44ab-b841-e4eea224fded" (UID: "f9008e53-2fc7-44ab-b841-e4eea224fded"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954447 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9008e53-2fc7-44ab-b841-e4eea224fded-istiod-ca-cert" pod "f9008e53-2fc7-44ab-b841-e4eea224fded" (UID: "f9008e53-2fc7-44ab-b841-e4eea224fded"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954453 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-credential-socket\") pod \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954525 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-kube-api-access-9td42\" (UniqueName: \"kubernetes.io/projected/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-kube-api-access-9td42\") pod \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954701 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-workload-certs" pod "25c9e11e-b82c-4e33-81bb-1ccae0588c2d" (UID: "25c9e11e-b82c-4e33-81bb-1ccae0588c2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954729 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-credential-socket" pod "25c9e11e-b82c-4e33-81bb-1ccae0588c2d" (UID: "25c9e11e-b82c-4e33-81bb-1ccae0588c2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954731 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istiod-ca-cert\") pod \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954789 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-token\" (UniqueName: \"kubernetes.io/projected/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-token\") pod \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954831 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-podinfo\") pod \"f9008e53-2fc7-44ab-b841-e4eea224fded\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954921 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-workload-certs" pod "f9008e53-2fc7-44ab-b841-e4eea224fded" (UID: "f9008e53-2fc7-44ab-b841-e4eea224fded"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954945 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-podinfo\") pod \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.954988 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-workload-certs\") pod \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.955013 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-kube-api-access-ncwsw\" (UniqueName: \"kubernetes.io/projected/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-kube-api-access-ncwsw\") pod \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.955016 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istiod-ca-cert" pod "25c9e11e-b82c-4e33-81bb-1ccae0588c2d" (UID: "25c9e11e-b82c-4e33-81bb-1ccae0588c2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.955050 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-envoy\") pod \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.955095 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-workload-socket\") pod \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.955137 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-podinfo\") pod \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\" (UID: \"25c9e11e-b82c-4e33-81bb-1ccae0588c2d\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.955169 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-credential-socket\") pod \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.955192 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istiod-ca-cert\") pod \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\" (UID: \"b7d46753-0b7a-4c8c-9870-31cedadbe4c5\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.955228 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-data\") pod \"f9008e53-2fc7-44ab-b841-e4eea224fded\" (UID: \"f9008e53-2fc7-44ab-b841-e4eea224fded\") " Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.955821 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9008e53-2fc7-44ab-b841-e4eea224fded-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.955838 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.955848 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.955856 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.955865 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.955874 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.955883 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.955894 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:33.956154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.955902 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:33.957997 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.957225 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-workload-socket" pod "f9008e53-2fc7-44ab-b841-e4eea224fded" (UID: "f9008e53-2fc7-44ab-b841-e4eea224fded"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:33.957997 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.957902 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-kube-api-access-9td42" pod "25c9e11e-b82c-4e33-81bb-1ccae0588c2d" (UID: "25c9e11e-b82c-4e33-81bb-1ccae0588c2d"). InnerVolumeSpecName "kube-api-access-9td42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:54:33.958185 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.958156 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-data" pod "f9008e53-2fc7-44ab-b841-e4eea224fded" (UID: "f9008e53-2fc7-44ab-b841-e4eea224fded"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:33.958393 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.958375 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-credential-socket" pod "b7d46753-0b7a-4c8c-9870-31cedadbe4c5" (UID: "b7d46753-0b7a-4c8c-9870-31cedadbe4c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:33.958618 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.958600 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istiod-ca-cert" pod "b7d46753-0b7a-4c8c-9870-31cedadbe4c5" (UID: "b7d46753-0b7a-4c8c-9870-31cedadbe4c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:54:33.958831 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.958811 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-workload-socket" pod "25c9e11e-b82c-4e33-81bb-1ccae0588c2d" (UID: "25c9e11e-b82c-4e33-81bb-1ccae0588c2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:33.959399 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.959373 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9008e53-2fc7-44ab-b841-e4eea224fded-kube-api-access-lsk4w" pod "f9008e53-2fc7-44ab-b841-e4eea224fded" (UID: "f9008e53-2fc7-44ab-b841-e4eea224fded"). InnerVolumeSpecName "kube-api-access-lsk4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:54:33.959502 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.959449 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-token" pod "f9008e53-2fc7-44ab-b841-e4eea224fded" (UID: "f9008e53-2fc7-44ab-b841-e4eea224fded"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:54:33.959627 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.959601 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-token" pod "25c9e11e-b82c-4e33-81bb-1ccae0588c2d" (UID: "25c9e11e-b82c-4e33-81bb-1ccae0588c2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:54:33.960662 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.960633 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-envoy" pod "25c9e11e-b82c-4e33-81bb-1ccae0588c2d" (UID: "25c9e11e-b82c-4e33-81bb-1ccae0588c2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:33.965890 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.960843 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-workload-certs" pod "b7d46753-0b7a-4c8c-9870-31cedadbe4c5" (UID: "b7d46753-0b7a-4c8c-9870-31cedadbe4c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:33.965890 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.961610 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-envoy" pod "b7d46753-0b7a-4c8c-9870-31cedadbe4c5" (UID: "b7d46753-0b7a-4c8c-9870-31cedadbe4c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:33.965890 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.961744 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-podinfo" pod "25c9e11e-b82c-4e33-81bb-1ccae0588c2d" (UID: "25c9e11e-b82c-4e33-81bb-1ccae0588c2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:54:33.965890 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.961830 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-envoy" pod "f9008e53-2fc7-44ab-b841-e4eea224fded" (UID: "f9008e53-2fc7-44ab-b841-e4eea224fded"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:33.965890 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.962123 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-kube-api-access-ncwsw" pod "b7d46753-0b7a-4c8c-9870-31cedadbe4c5" (UID: "b7d46753-0b7a-4c8c-9870-31cedadbe4c5"). InnerVolumeSpecName "kube-api-access-ncwsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:54:33.967002 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.966962 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-podinfo" pod "b7d46753-0b7a-4c8c-9870-31cedadbe4c5" (UID: "b7d46753-0b7a-4c8c-9870-31cedadbe4c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:54:33.967100 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.967049 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-podinfo" pod "f9008e53-2fc7-44ab-b841-e4eea224fded" (UID: "f9008e53-2fc7-44ab-b841-e4eea224fded"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:54:33.967527 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:33.967486 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-token" pod "b7d46753-0b7a-4c8c-9870-31cedadbe4c5" (UID: "b7d46753-0b7a-4c8c-9870-31cedadbe4c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:54:34.056885 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.056839 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98be750b-4e74-4378-871b-3a3e9d53cd78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98be750b-4e74-4378-871b-3a3e9d53cd78-istiod-ca-cert\") pod \"98be750b-4e74-4378-871b-3a3e9d53cd78\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " Nov 02 01:54:34.057105 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.056944 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-podinfo\") pod \"98be750b-4e74-4378-871b-3a3e9d53cd78\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " Nov 02 01:54:34.057105 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.057000 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-envoy\") pod \"98be750b-4e74-4378-871b-3a3e9d53cd78\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " Nov 02 01:54:34.057105 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.057042 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-token\" (UniqueName: \"kubernetes.io/projected/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-token\") pod \"98be750b-4e74-4378-871b-3a3e9d53cd78\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " Nov 02 01:54:34.057722 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.057164 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-workload-certs\") pod \"98be750b-4e74-4378-871b-3a3e9d53cd78\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " Nov 02 01:54:34.057722 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.057246 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98be750b-4e74-4378-871b-3a3e9d53cd78-kube-api-access-ph7d9\" (UniqueName: \"kubernetes.io/projected/98be750b-4e74-4378-871b-3a3e9d53cd78-kube-api-access-ph7d9\") pod \"98be750b-4e74-4378-871b-3a3e9d53cd78\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " Nov 02 01:54:34.057722 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.057291 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98be750b-4e74-4378-871b-3a3e9d53cd78-istiod-ca-cert" pod "98be750b-4e74-4378-871b-3a3e9d53cd78" (UID: "98be750b-4e74-4378-871b-3a3e9d53cd78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:54:34.057722 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.057326 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-workload-socket\") pod \"98be750b-4e74-4378-871b-3a3e9d53cd78\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " Nov 02 01:54:34.057722 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.057388 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-credential-socket\") pod \"98be750b-4e74-4378-871b-3a3e9d53cd78\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " Nov 02 01:54:34.057722 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.057476 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-data\") pod \"98be750b-4e74-4378-871b-3a3e9d53cd78\" (UID: \"98be750b-4e74-4378-871b-3a3e9d53cd78\") " Nov 02 01:54:34.057722 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.057480 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-workload-certs" pod "98be750b-4e74-4378-871b-3a3e9d53cd78" (UID: "98be750b-4e74-4378-871b-3a3e9d53cd78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:34.058045 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.057748 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-workload-socket" pod "98be750b-4e74-4378-871b-3a3e9d53cd78" (UID: "98be750b-4e74-4378-871b-3a3e9d53cd78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:34.058045 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.057851 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-data" pod "98be750b-4e74-4378-871b-3a3e9d53cd78" (UID: "98be750b-4e74-4378-871b-3a3e9d53cd78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:34.058045 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.057869 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-credential-socket" pod "98be750b-4e74-4378-871b-3a3e9d53cd78" (UID: "98be750b-4e74-4378-871b-3a3e9d53cd78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:34.058639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.058331 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.058639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.058359 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.058639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.058374 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.058639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.058388 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.058639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.058402 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.058639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.058416 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98be750b-4e74-4378-871b-3a3e9d53cd78-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.058639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.058431 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.058639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.058445 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsk4w\" (UniqueName: \"kubernetes.io/projected/f9008e53-2fc7-44ab-b841-e4eea224fded-kube-api-access-lsk4w\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.058639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.058459 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.058639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.058472 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.058639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.058486 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9td42\" (UniqueName: \"kubernetes.io/projected/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-kube-api-access-9td42\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.058639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.058500 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.058639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.058513 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.058639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.058527 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.058639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.058541 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.058639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.058556 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncwsw\" (UniqueName: \"kubernetes.io/projected/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-kube-api-access-ncwsw\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.058639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.058569 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.058639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.058586 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.058639 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.058600 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.059806 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.059727 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25c9e11e-b82c-4e33-81bb-1ccae0588c2d-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.059806 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.059759 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.059806 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.059770 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d46753-0b7a-4c8c-9870-31cedadbe4c5-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.059806 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.059782 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9008e53-2fc7-44ab-b841-e4eea224fded-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.060069 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.059923 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-podinfo" pod "98be750b-4e74-4378-871b-3a3e9d53cd78" (UID: "98be750b-4e74-4378-871b-3a3e9d53cd78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:54:34.060069 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.059941 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-token" pod "98be750b-4e74-4378-871b-3a3e9d53cd78" (UID: "98be750b-4e74-4378-871b-3a3e9d53cd78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:54:34.060069 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.060011 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-envoy" pod "98be750b-4e74-4378-871b-3a3e9d53cd78" (UID: "98be750b-4e74-4378-871b-3a3e9d53cd78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:34.062079 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.062053 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98be750b-4e74-4378-871b-3a3e9d53cd78-kube-api-access-ph7d9" pod "98be750b-4e74-4378-871b-3a3e9d53cd78" (UID: "98be750b-4e74-4378-871b-3a3e9d53cd78"). InnerVolumeSpecName "kube-api-access-ph7d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:54:34.161351 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.161301 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.161351 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.161344 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.161351 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.161360 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98be750b-4e74-4378-871b-3a3e9d53cd78-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.161586 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.161372 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph7d9\" (UniqueName: \"kubernetes.io/projected/98be750b-4e74-4378-871b-3a3e9d53cd78-kube-api-access-ph7d9\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:34.710919 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.710833 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:34.802565 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.802531 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" Nov 02 01:54:34.803083 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.802573 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f" event={"ID":"98be750b-4e74-4378-871b-3a3e9d53cd78","Type":"ContainerDied","Data":"92348f2eee7c3343f10cea8c5778625b737536301eadce8f439cb5ef0449ca59"} Nov 02 01:54:34.803083 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.802635 5044 scope.go:122] "RemoveContainer" containerID="f49c280fc18c14ccfa7c4c3c659dabc621db819e2a3200589db9c35475e37122" Nov 02 01:54:34.816368 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.816320 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" event={"ID":"25c9e11e-b82c-4e33-81bb-1ccae0588c2d","Type":"ContainerDied","Data":"169fed294880ec6f0defd2876310bd0e2e6430e9965c165318a90c8677694b62"} Nov 02 01:54:34.816517 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.816503 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw" Nov 02 01:54:34.821417 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.821387 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657" Nov 02 01:54:34.827097 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.827070 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" event={"ID":"f9008e53-2fc7-44ab-b841-e4eea224fded","Type":"ContainerDied","Data":"7ba41ef97db99158553eea634297cbba807efdcf3d3e331c60d75ea8d9527065"} Nov 02 01:54:34.827206 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.827088 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559" Nov 02 01:54:34.827394 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.827374 5044 scope.go:122] "RemoveContainer" containerID="84ca97d6aef5c1718e897f117e436a737109df42e4486966e9497f3dc89d1b46" Nov 02 01:54:34.841246 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.841220 5044 scope.go:122] "RemoveContainer" containerID="89ef6add7905cab77444f3b6a5a42d0e02c466bb33e76e4b7fc9a80138f82426" Nov 02 01:54:34.977167 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:34.977083 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f"] Nov 02 01:54:35.027136 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:35.027104 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p4v5f"] Nov 02 01:54:35.098938 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:35.098908 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw"] Nov 02 01:54:35.104626 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:35.104599 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-p2nmw"] Nov 02 01:54:35.284837 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:35.284802 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657"] Nov 02 01:54:35.285045 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:35.285013 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:35.291609 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:35.291585 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-qf657"] Nov 02 01:54:35.401412 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:35.401379 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559"] Nov 02 01:54:35.406868 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:35.406840 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5888dc-openshift-default-749d8fbcf7-2z559"] Nov 02 01:54:35.458526 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:35.458493 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25c9e11e-b82c-4e33-81bb-1ccae0588c2d" path="/var/lib/kubelet/pods/25c9e11e-b82c-4e33-81bb-1ccae0588c2d/volumes" Nov 02 01:54:35.459118 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:35.459100 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98be750b-4e74-4378-871b-3a3e9d53cd78" path="/var/lib/kubelet/pods/98be750b-4e74-4378-871b-3a3e9d53cd78/volumes" Nov 02 01:54:35.459573 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:35.459559 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7d46753-0b7a-4c8c-9870-31cedadbe4c5" path="/var/lib/kubelet/pods/b7d46753-0b7a-4c8c-9870-31cedadbe4c5/volumes" Nov 02 01:54:35.460402 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:35.460385 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9008e53-2fc7-44ab-b841-e4eea224fded" path="/var/lib/kubelet/pods/f9008e53-2fc7-44ab-b841-e4eea224fded/volumes" Nov 02 01:54:35.658721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:35.658576 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:35.895914 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:35.895816 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:36.343143 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:36.343110 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:40.271738 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:40.271702 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:42.569704 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:42.568235 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc"] Nov 02 01:54:42.569704 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:42.568484 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" podUID="44289fed-9af9-4b30-86f2-c4ec83988f44" containerName="istio-proxy" containerID="cri-o://393f689d7f563170c7ff9b2fc65c2becabb7f41647ae0ad340935aaf4fce2c4e" gracePeriod=30 Nov 02 01:54:42.960830 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:42.904437 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:43.673227 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:43.673198 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 01:54:43.677846 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:43.677827 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 01:54:44.002615 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:44.002532 5044 scope.go:122] "RemoveContainer" containerID="8e926e2b9a3d686c91cc707f8d22553cb53479e5203ae105f3089448f8ecf582" Nov 02 01:54:44.014084 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:44.014061 5044 scope.go:122] "RemoveContainer" containerID="215d1088a6457e2da21f285503ff09dca44db2d4b7eda1d4c13e2afff4a6915b" Nov 02 01:54:44.024347 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:44.024324 5044 scope.go:122] "RemoveContainer" containerID="97e5795c2a597e4457a9a4b870dd042539975dcb1ded9e759f5977fcd0f9c6d6" Nov 02 01:54:44.032903 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:44.032871 5044 scope.go:122] "RemoveContainer" containerID="5250a703d6cd7d9dbb8a3645fd78be98119f4bca728f658dfa68a3f29d312f2f" Nov 02 01:54:44.042037 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:44.042016 5044 scope.go:122] "RemoveContainer" containerID="0a5238cf38d8ce14ca021378490b2a9eb0456f9e5fb5ede7ebb05f38728aee47" Nov 02 01:54:44.544652 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:44.544613 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:45.546867 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.546835 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4"] Nov 02 01:54:45.547543 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.547528 5044 cpu_manager.go:414] "containerMap: removing container" podUID="b7d46753-0b7a-4c8c-9870-31cedadbe4c5" containerName="istio-proxy" Nov 02 01:54:45.547543 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.547544 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d46753-0b7a-4c8c-9870-31cedadbe4c5" containerName="istio-proxy" Nov 02 01:54:45.547636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.547556 5044 cpu_manager.go:414] "containerMap: removing container" podUID="f9008e53-2fc7-44ab-b841-e4eea224fded" containerName="istio-proxy" Nov 02 01:54:45.547636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.547562 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9008e53-2fc7-44ab-b841-e4eea224fded" containerName="istio-proxy" Nov 02 01:54:45.547636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.547594 5044 cpu_manager.go:414] "containerMap: removing container" podUID="98be750b-4e74-4378-871b-3a3e9d53cd78" containerName="istio-proxy" Nov 02 01:54:45.547636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.547598 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98be750b-4e74-4378-871b-3a3e9d53cd78" containerName="istio-proxy" Nov 02 01:54:45.547636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.547608 5044 cpu_manager.go:414] "containerMap: removing container" podUID="b7d46753-0b7a-4c8c-9870-31cedadbe4c5" containerName="istio-proxy" Nov 02 01:54:45.547636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.547612 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d46753-0b7a-4c8c-9870-31cedadbe4c5" containerName="istio-proxy" Nov 02 01:54:45.547636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.547623 5044 cpu_manager.go:414] "containerMap: removing container" podUID="25c9e11e-b82c-4e33-81bb-1ccae0588c2d" containerName="istio-proxy" Nov 02 01:54:45.547636 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.547628 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25c9e11e-b82c-4e33-81bb-1ccae0588c2d" containerName="istio-proxy" Nov 02 01:54:45.547880 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.547790 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d46753-0b7a-4c8c-9870-31cedadbe4c5" containerName="istio-proxy" Nov 02 01:54:45.547880 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.547810 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9008e53-2fc7-44ab-b841-e4eea224fded" containerName="istio-proxy" Nov 02 01:54:45.547880 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.547825 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="98be750b-4e74-4378-871b-3a3e9d53cd78" containerName="istio-proxy" Nov 02 01:54:45.547880 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.547837 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="25c9e11e-b82c-4e33-81bb-1ccae0588c2d" containerName="istio-proxy" Nov 02 01:54:45.554986 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.554968 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" Nov 02 01:54:45.609619 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.609590 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4"] Nov 02 01:54:45.655497 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.655462 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-07ac70d0-serving-cert\"" Nov 02 01:54:45.660430 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.660393 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc8j5\" (UniqueName: \"kubernetes.io/projected/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-kube-api-access-lc8j5\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4\" (UID: \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" Nov 02 01:54:45.660627 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.660599 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-maas-api-tls\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4\" (UID: \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" Nov 02 01:54:45.660856 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.660836 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-openshift-service-ca\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4\" (UID: \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" Nov 02 01:54:45.660970 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.660950 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-metrics-certs\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4\" (UID: \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" Nov 02 01:54:45.746633 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.746599 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:45.762283 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.762256 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-maas-api-tls\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4\" (UID: \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" Nov 02 01:54:45.762427 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.762377 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-openshift-service-ca\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4\" (UID: \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" Nov 02 01:54:45.762427 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:54:45.762395 5044 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-serving-cert: secret "maas-api-e2e-mt-07ac70d0-serving-cert" not found Nov 02 01:54:45.762527 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.762430 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-metrics-certs\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4\" (UID: \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" Nov 02 01:54:45.762527 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:54:45.762462 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-maas-api-tls podName:66b4726e-ea0f-46e5-b1a8-cf803f344ef4 nodeName:}" failed. No retries permitted until 2026-11-02 01:54:46.262442907 +0000 UTC m=+1802.933695566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-maas-api-tls") pod "maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" (UID: "66b4726e-ea0f-46e5-b1a8-cf803f344ef4") : secret "maas-api-e2e-mt-07ac70d0-serving-cert" not found Nov 02 01:54:45.762692 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.762572 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc8j5\" (UniqueName: \"kubernetes.io/projected/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-kube-api-access-lc8j5\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4\" (UID: \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" Nov 02 01:54:45.763126 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.763102 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-openshift-service-ca\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4\" (UID: \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" Nov 02 01:54:45.765014 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.764993 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-metrics-certs\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4\" (UID: \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" Nov 02 01:54:45.830635 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.830549 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc8j5\" (UniqueName: \"kubernetes.io/projected/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-kube-api-access-lc8j5\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4\" (UID: \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" Nov 02 01:54:45.917944 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:45.917916 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:46.063984 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:46.063950 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:46.287238 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:46.287186 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-maas-api-tls\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4\" (UID: \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" Nov 02 01:54:46.289679 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:46.289647 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-maas-api-tls\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4\" (UID: \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" Nov 02 01:54:46.472607 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:46.472572 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" Nov 02 01:54:46.862541 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:54:46.862274 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b4726e_ea0f_46e5_b1a8_cf803f344ef4.slice/crio-a071bd1857c6712db764f9081508993068f9e0e2ed6ccc2b7f89da9c072f8798 WatchSource:0}: Error finding container a071bd1857c6712db764f9081508993068f9e0e2ed6ccc2b7f89da9c072f8798: Status 404 returned error can't find the container with id a071bd1857c6712db764f9081508993068f9e0e2ed6ccc2b7f89da9c072f8798 Nov 02 01:54:46.862541 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:46.862426 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4"] Nov 02 01:54:46.864541 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:46.864519 5044 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 01:54:47.124129 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:47.124044 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" event={"ID":"66b4726e-ea0f-46e5-b1a8-cf803f344ef4","Type":"ContainerStarted","Data":"a071bd1857c6712db764f9081508993068f9e0e2ed6ccc2b7f89da9c072f8798"} Nov 02 01:54:47.885660 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:47.885636 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:54:48.010067 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.010043 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44289fed-9af9-4b30-86f2-c4ec83988f44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44289fed-9af9-4b30-86f2-c4ec83988f44-istiod-ca-cert\") pod \"44289fed-9af9-4b30-86f2-c4ec83988f44\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " Nov 02 01:54:48.010187 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.010080 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-workload-certs\") pod \"44289fed-9af9-4b30-86f2-c4ec83988f44\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " Nov 02 01:54:48.010187 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.010163 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-workload-socket\") pod \"44289fed-9af9-4b30-86f2-c4ec83988f44\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " Nov 02 01:54:48.010323 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.010218 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-data\") pod \"44289fed-9af9-4b30-86f2-c4ec83988f44\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " Nov 02 01:54:48.010407 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.010387 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-token\" (UniqueName: \"kubernetes.io/projected/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-token\") pod \"44289fed-9af9-4b30-86f2-c4ec83988f44\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " Nov 02 01:54:48.010507 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.010487 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-workload-certs" pod "44289fed-9af9-4b30-86f2-c4ec83988f44" (UID: "44289fed-9af9-4b30-86f2-c4ec83988f44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:48.010572 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.010503 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44289fed-9af9-4b30-86f2-c4ec83988f44-istiod-ca-cert" pod "44289fed-9af9-4b30-86f2-c4ec83988f44" (UID: "44289fed-9af9-4b30-86f2-c4ec83988f44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:54:48.010572 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.010509 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-podinfo\") pod \"44289fed-9af9-4b30-86f2-c4ec83988f44\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " Nov 02 01:54:48.010703 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.010589 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44289fed-9af9-4b30-86f2-c4ec83988f44-kube-api-access-sbtfw\" (UniqueName: \"kubernetes.io/projected/44289fed-9af9-4b30-86f2-c4ec83988f44-kube-api-access-sbtfw\") pod \"44289fed-9af9-4b30-86f2-c4ec83988f44\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " Nov 02 01:54:48.010703 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.010622 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-envoy\") pod \"44289fed-9af9-4b30-86f2-c4ec83988f44\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " Nov 02 01:54:48.010703 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.010622 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-data" pod "44289fed-9af9-4b30-86f2-c4ec83988f44" (UID: "44289fed-9af9-4b30-86f2-c4ec83988f44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:48.010703 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.010665 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-credential-socket\") pod \"44289fed-9af9-4b30-86f2-c4ec83988f44\" (UID: \"44289fed-9af9-4b30-86f2-c4ec83988f44\") " Nov 02 01:54:48.011082 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.011047 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-workload-socket" pod "44289fed-9af9-4b30-86f2-c4ec83988f44" (UID: "44289fed-9af9-4b30-86f2-c4ec83988f44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:48.011568 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.011543 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-credential-socket" pod "44289fed-9af9-4b30-86f2-c4ec83988f44" (UID: "44289fed-9af9-4b30-86f2-c4ec83988f44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:48.011923 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.011898 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:48.012026 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.011925 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44289fed-9af9-4b30-86f2-c4ec83988f44-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:48.012026 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.011939 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:48.012026 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.011955 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:48.012026 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.011969 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:48.013106 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.012901 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-podinfo" pod "44289fed-9af9-4b30-86f2-c4ec83988f44" (UID: "44289fed-9af9-4b30-86f2-c4ec83988f44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:54:48.013555 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.013529 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-envoy" pod "44289fed-9af9-4b30-86f2-c4ec83988f44" (UID: "44289fed-9af9-4b30-86f2-c4ec83988f44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:54:48.013888 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.013867 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-token" pod "44289fed-9af9-4b30-86f2-c4ec83988f44" (UID: "44289fed-9af9-4b30-86f2-c4ec83988f44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:54:48.014553 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.014532 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44289fed-9af9-4b30-86f2-c4ec83988f44-kube-api-access-sbtfw" pod "44289fed-9af9-4b30-86f2-c4ec83988f44" (UID: "44289fed-9af9-4b30-86f2-c4ec83988f44"). InnerVolumeSpecName "kube-api-access-sbtfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:54:48.113623 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.113546 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:48.113623 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.113580 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:48.113623 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.113596 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbtfw\" (UniqueName: \"kubernetes.io/projected/44289fed-9af9-4b30-86f2-c4ec83988f44-kube-api-access-sbtfw\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:48.113623 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.113610 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44289fed-9af9-4b30-86f2-c4ec83988f44-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:54:48.134034 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.133997 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" event={"ID":"66b4726e-ea0f-46e5-b1a8-cf803f344ef4","Type":"ContainerStarted","Data":"d50fa6a11b74a0ce2308f765cd5b94f65d52fcfddaa41e3ce5df232226499c32"} Nov 02 01:54:48.134894 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.134868 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" Nov 02 01:54:48.137211 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.137182 5044 generic.go:356] "Generic (PLEG): container finished" podID="44289fed-9af9-4b30-86f2-c4ec83988f44" containerID="393f689d7f563170c7ff9b2fc65c2becabb7f41647ae0ad340935aaf4fce2c4e" exitCode=0 Nov 02 01:54:48.137335 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.137217 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" Nov 02 01:54:48.137335 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.137255 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" event={"ID":"44289fed-9af9-4b30-86f2-c4ec83988f44","Type":"ContainerDied","Data":"393f689d7f563170c7ff9b2fc65c2becabb7f41647ae0ad340935aaf4fce2c4e"} Nov 02 01:54:48.137335 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.137305 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc" event={"ID":"44289fed-9af9-4b30-86f2-c4ec83988f44","Type":"ContainerDied","Data":"87dfe53f35864f0d0872a6b6897049dd0026c6163185735950e170dc2ee89eb7"} Nov 02 01:54:48.137335 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.137334 5044 scope.go:122] "RemoveContainer" containerID="393f689d7f563170c7ff9b2fc65c2becabb7f41647ae0ad340935aaf4fce2c4e" Nov 02 01:54:48.151184 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.151164 5044 scope.go:122] "RemoveContainer" containerID="393f689d7f563170c7ff9b2fc65c2becabb7f41647ae0ad340935aaf4fce2c4e" Nov 02 01:54:48.151491 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:54:48.151464 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"393f689d7f563170c7ff9b2fc65c2becabb7f41647ae0ad340935aaf4fce2c4e\": container with ID starting with 393f689d7f563170c7ff9b2fc65c2becabb7f41647ae0ad340935aaf4fce2c4e not found: ID does not exist" containerID="393f689d7f563170c7ff9b2fc65c2becabb7f41647ae0ad340935aaf4fce2c4e" Nov 02 01:54:48.151548 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.151503 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"393f689d7f563170c7ff9b2fc65c2becabb7f41647ae0ad340935aaf4fce2c4e"} err="failed to get container status \"393f689d7f563170c7ff9b2fc65c2becabb7f41647ae0ad340935aaf4fce2c4e\": rpc error: code = NotFound desc = could not find container \"393f689d7f563170c7ff9b2fc65c2becabb7f41647ae0ad340935aaf4fce2c4e\": container with ID starting with 393f689d7f563170c7ff9b2fc65c2becabb7f41647ae0ad340935aaf4fce2c4e not found: ID does not exist" Nov 02 01:54:48.255464 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.255415 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" podStartSLOduration=2.1622779850000002 podStartE2EDuration="3.255396415s" totalImagesPullingTime="1.09311843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:48.253652726 +0000 UTC m=+1804.924905408" watchObservedRunningTime="2026-11-02 01:54:48.255396415 +0000 UTC m=+1804.926649111" Nov 02 01:54:48.392005 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.391915 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc"] Nov 02 01:54:48.396269 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:48.396234 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-46dgc"] Nov 02 01:54:49.453828 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:49.453793 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44289fed-9af9-4b30-86f2-c4ec83988f44" path="/var/lib/kubelet/pods/44289fed-9af9-4b30-86f2-c4ec83988f44/volumes" Nov 02 01:54:50.797802 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:50.797767 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:51.403392 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:51.403361 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:51.583223 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:51.583138 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:54.184133 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:54:54.184091 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:54:55.159724 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.159663 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" Nov 02 01:54:55.573844 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.573812 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd"] Nov 02 01:54:55.574777 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.574758 5044 cpu_manager.go:414] "containerMap: removing container" podUID="44289fed-9af9-4b30-86f2-c4ec83988f44" containerName="istio-proxy" Nov 02 01:54:55.574831 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.574780 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44289fed-9af9-4b30-86f2-c4ec83988f44" containerName="istio-proxy" Nov 02 01:54:55.575040 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.575027 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d46753-0b7a-4c8c-9870-31cedadbe4c5" containerName="istio-proxy" Nov 02 01:54:55.575073 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.575048 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="44289fed-9af9-4b30-86f2-c4ec83988f44" containerName="istio-proxy" Nov 02 01:54:55.580824 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.580803 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" Nov 02 01:54:55.634177 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.634146 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-257466-serving-cert\"" Nov 02 01:54:55.635859 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.635827 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd"] Nov 02 01:54:55.770562 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.770520 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4052c546-2071-483f-878f-290e59139928-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd\" (UID: \"4052c546-2071-483f-878f-290e59139928\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" Nov 02 01:54:55.770813 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.770592 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bbh8\" (UniqueName: \"kubernetes.io/projected/4052c546-2071-483f-878f-290e59139928-kube-api-access-8bbh8\") pod \"maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd\" (UID: \"4052c546-2071-483f-878f-290e59139928\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" Nov 02 01:54:55.770883 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.770801 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4052c546-2071-483f-878f-290e59139928-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd\" (UID: \"4052c546-2071-483f-878f-290e59139928\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" Nov 02 01:54:55.770883 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.770857 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4052c546-2071-483f-878f-290e59139928-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd\" (UID: \"4052c546-2071-483f-878f-290e59139928\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" Nov 02 01:54:55.826472 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.826372 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:55.872865 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.872827 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4052c546-2071-483f-878f-290e59139928-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd\" (UID: \"4052c546-2071-483f-878f-290e59139928\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" Nov 02 01:54:55.873066 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.872897 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4052c546-2071-483f-878f-290e59139928-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd\" (UID: \"4052c546-2071-483f-878f-290e59139928\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" Nov 02 01:54:55.873066 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:54:55.872999 5044 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-serving-cert: secret "maas-api-e2e-shared-a-w5-257466-serving-cert" not found Nov 02 01:54:55.873171 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:54:55.873094 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4052c546-2071-483f-878f-290e59139928-maas-api-tls podName:4052c546-2071-483f-878f-290e59139928 nodeName:}" failed. No retries permitted until 2026-11-02 01:54:56.373070325 +0000 UTC m=+1813.044322990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4052c546-2071-483f-878f-290e59139928-maas-api-tls") pod "maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" (UID: "4052c546-2071-483f-878f-290e59139928") : secret "maas-api-e2e-shared-a-w5-257466-serving-cert" not found Nov 02 01:54:55.873237 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.873167 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4052c546-2071-483f-878f-290e59139928-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd\" (UID: \"4052c546-2071-483f-878f-290e59139928\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" Nov 02 01:54:55.873293 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.873244 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbh8\" (UniqueName: \"kubernetes.io/projected/4052c546-2071-483f-878f-290e59139928-kube-api-access-8bbh8\") pod \"maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd\" (UID: \"4052c546-2071-483f-878f-290e59139928\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" Nov 02 01:54:55.873730 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.873704 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4052c546-2071-483f-878f-290e59139928-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd\" (UID: \"4052c546-2071-483f-878f-290e59139928\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" Nov 02 01:54:55.876031 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.876008 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4052c546-2071-483f-878f-290e59139928-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd\" (UID: \"4052c546-2071-483f-878f-290e59139928\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" Nov 02 01:54:55.936719 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.936654 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bbh8\" (UniqueName: \"kubernetes.io/projected/4052c546-2071-483f-878f-290e59139928-kube-api-access-8bbh8\") pod \"maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd\" (UID: \"4052c546-2071-483f-878f-290e59139928\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" Nov 02 01:54:55.975971 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:55.975898 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:56.178960 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:56.178878 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:54:56.381503 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:56.381464 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4052c546-2071-483f-878f-290e59139928-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd\" (UID: \"4052c546-2071-483f-878f-290e59139928\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" Nov 02 01:54:56.384027 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:56.383997 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4052c546-2071-483f-878f-290e59139928-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd\" (UID: \"4052c546-2071-483f-878f-290e59139928\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" Nov 02 01:54:56.498528 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:56.498454 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" Nov 02 01:54:56.889374 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:56.889339 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd"] Nov 02 01:54:56.890808 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:54:56.890773 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4052c546_2071_483f_878f_290e59139928.slice/crio-f7989f02f1a03fd4d4fd5d8d67c54223213cf11a8e6e6b0240be244e30b5e629 WatchSource:0}: Error finding container f7989f02f1a03fd4d4fd5d8d67c54223213cf11a8e6e6b0240be244e30b5e629: Status 404 returned error can't find the container with id f7989f02f1a03fd4d4fd5d8d67c54223213cf11a8e6e6b0240be244e30b5e629 Nov 02 01:54:57.219265 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:57.219183 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" event={"ID":"4052c546-2071-483f-878f-290e59139928","Type":"ContainerStarted","Data":"f7989f02f1a03fd4d4fd5d8d67c54223213cf11a8e6e6b0240be244e30b5e629"} Nov 02 01:54:58.227739 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:58.227616 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" event={"ID":"4052c546-2071-483f-878f-290e59139928","Type":"ContainerStarted","Data":"9b48f02f3793dc25d3230879f20b44f4cb38942dd7d3b965749c66e79b51bc12"} Nov 02 01:54:58.227739 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:58.227725 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" Nov 02 01:54:58.347460 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:54:58.347407 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" podStartSLOduration=2.407289529 podStartE2EDuration="3.347392459s" totalImagesPullingTime="940.10293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:54:58.346123095 +0000 UTC m=+1815.017375778" watchObservedRunningTime="2026-11-02 01:54:58.347392459 +0000 UTC m=+1815.018645134" Nov 02 01:55:00.872828 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:00.872790 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:01.425984 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:01.425951 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:01.645708 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:01.635445 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:04.241996 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:04.241963 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" Nov 02 01:55:05.877947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:05.877913 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:10.673770 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:10.673733 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:13.146699 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:13.146650 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j"] Nov 02 01:55:13.147115 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:13.146878 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" podUID="a6523f25-d7c1-4881-8adc-788c8226ea9c" containerName="istio-proxy" containerID="cri-o://43bb1fa53265bd08f4efa69a428409a731aa7bd49866512abd2b119d08c0c711" gracePeriod=30 Nov 02 01:55:15.513014 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:15.512947 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:18.281537 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.281508 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 01:55:18.281941 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.281576 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" podUID="a6523f25-d7c1-4881-8adc-788c8226ea9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 01:55:18.439634 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.439604 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:55:18.461767 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.461740 5044 generic.go:356] "Generic (PLEG): container finished" podID="a6523f25-d7c1-4881-8adc-788c8226ea9c" containerID="43bb1fa53265bd08f4efa69a428409a731aa7bd49866512abd2b119d08c0c711" exitCode=0 Nov 02 01:55:18.461901 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.461783 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" Nov 02 01:55:18.461901 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.461837 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" event={"ID":"a6523f25-d7c1-4881-8adc-788c8226ea9c","Type":"ContainerDied","Data":"43bb1fa53265bd08f4efa69a428409a731aa7bd49866512abd2b119d08c0c711"} Nov 02 01:55:18.461901 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.461894 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j" event={"ID":"a6523f25-d7c1-4881-8adc-788c8226ea9c","Type":"ContainerDied","Data":"2315f423a0826bdc5307134017f63b4d3f14d510c6c70ce30f503af347cc2711"} Nov 02 01:55:18.462024 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.461923 5044 scope.go:122] "RemoveContainer" containerID="43bb1fa53265bd08f4efa69a428409a731aa7bd49866512abd2b119d08c0c711" Nov 02 01:55:18.472492 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.472471 5044 scope.go:122] "RemoveContainer" containerID="43bb1fa53265bd08f4efa69a428409a731aa7bd49866512abd2b119d08c0c711" Nov 02 01:55:18.472796 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:55:18.472775 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43bb1fa53265bd08f4efa69a428409a731aa7bd49866512abd2b119d08c0c711\": container with ID starting with 43bb1fa53265bd08f4efa69a428409a731aa7bd49866512abd2b119d08c0c711 not found: ID does not exist" containerID="43bb1fa53265bd08f4efa69a428409a731aa7bd49866512abd2b119d08c0c711" Nov 02 01:55:18.472881 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.472803 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43bb1fa53265bd08f4efa69a428409a731aa7bd49866512abd2b119d08c0c711"} err="failed to get container status \"43bb1fa53265bd08f4efa69a428409a731aa7bd49866512abd2b119d08c0c711\": rpc error: code = NotFound desc = could not find container \"43bb1fa53265bd08f4efa69a428409a731aa7bd49866512abd2b119d08c0c711\": container with ID starting with 43bb1fa53265bd08f4efa69a428409a731aa7bd49866512abd2b119d08c0c711 not found: ID does not exist" Nov 02 01:55:18.536889 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.536857 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-envoy\") pod \"a6523f25-d7c1-4881-8adc-788c8226ea9c\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " Nov 02 01:55:18.537027 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.536975 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6523f25-d7c1-4881-8adc-788c8226ea9c-kube-api-access-scm6l\" (UniqueName: \"kubernetes.io/projected/a6523f25-d7c1-4881-8adc-788c8226ea9c-kube-api-access-scm6l\") pod \"a6523f25-d7c1-4881-8adc-788c8226ea9c\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " Nov 02 01:55:18.537107 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.537087 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-podinfo\") pod \"a6523f25-d7c1-4881-8adc-788c8226ea9c\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " Nov 02 01:55:18.537178 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.537162 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-workload-socket\") pod \"a6523f25-d7c1-4881-8adc-788c8226ea9c\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " Nov 02 01:55:18.537246 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.537189 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-token\" (UniqueName: \"kubernetes.io/projected/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-token\") pod \"a6523f25-d7c1-4881-8adc-788c8226ea9c\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " Nov 02 01:55:18.537310 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.537272 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6523f25-d7c1-4881-8adc-788c8226ea9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6523f25-d7c1-4881-8adc-788c8226ea9c-istiod-ca-cert\") pod \"a6523f25-d7c1-4881-8adc-788c8226ea9c\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " Nov 02 01:55:18.537364 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.537339 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-credential-socket\") pod \"a6523f25-d7c1-4881-8adc-788c8226ea9c\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " Nov 02 01:55:18.537418 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.537376 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-data\") pod \"a6523f25-d7c1-4881-8adc-788c8226ea9c\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " Nov 02 01:55:18.537418 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.537395 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-workload-socket" pod "a6523f25-d7c1-4881-8adc-788c8226ea9c" (UID: "a6523f25-d7c1-4881-8adc-788c8226ea9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:55:18.537652 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.537632 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-workload-certs\") pod \"a6523f25-d7c1-4881-8adc-788c8226ea9c\" (UID: \"a6523f25-d7c1-4881-8adc-788c8226ea9c\") " Nov 02 01:55:18.537751 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.537646 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-credential-socket" pod "a6523f25-d7c1-4881-8adc-788c8226ea9c" (UID: "a6523f25-d7c1-4881-8adc-788c8226ea9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:55:18.537815 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.537765 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6523f25-d7c1-4881-8adc-788c8226ea9c-istiod-ca-cert" pod "a6523f25-d7c1-4881-8adc-788c8226ea9c" (UID: "a6523f25-d7c1-4881-8adc-788c8226ea9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:55:18.538414 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.537907 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-data" pod "a6523f25-d7c1-4881-8adc-788c8226ea9c" (UID: "a6523f25-d7c1-4881-8adc-788c8226ea9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:55:18.538938 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.538528 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:55:18.538938 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.538556 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6523f25-d7c1-4881-8adc-788c8226ea9c-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:55:18.538938 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.538571 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:55:18.538938 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.538586 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:55:18.538938 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.538740 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-workload-certs" pod "a6523f25-d7c1-4881-8adc-788c8226ea9c" (UID: "a6523f25-d7c1-4881-8adc-788c8226ea9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:55:18.539513 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.539490 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6523f25-d7c1-4881-8adc-788c8226ea9c-kube-api-access-scm6l" pod "a6523f25-d7c1-4881-8adc-788c8226ea9c" (UID: "a6523f25-d7c1-4881-8adc-788c8226ea9c"). InnerVolumeSpecName "kube-api-access-scm6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:55:18.539598 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.539511 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-envoy" pod "a6523f25-d7c1-4881-8adc-788c8226ea9c" (UID: "a6523f25-d7c1-4881-8adc-788c8226ea9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:55:18.539656 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.539641 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-podinfo" pod "a6523f25-d7c1-4881-8adc-788c8226ea9c" (UID: "a6523f25-d7c1-4881-8adc-788c8226ea9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:55:18.539917 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.539901 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-token" pod "a6523f25-d7c1-4881-8adc-788c8226ea9c" (UID: "a6523f25-d7c1-4881-8adc-788c8226ea9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:55:18.640046 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.640007 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:55:18.640046 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.640045 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:55:18.640295 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.640060 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:55:18.640295 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.640075 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6523f25-d7c1-4881-8adc-788c8226ea9c-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:55:18.640295 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.640089 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scm6l\" (UniqueName: \"kubernetes.io/projected/a6523f25-d7c1-4881-8adc-788c8226ea9c-kube-api-access-scm6l\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:55:18.893901 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.893868 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j"] Nov 02 01:55:18.897396 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:18.897368 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-6j95j"] Nov 02 01:55:19.453790 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:19.453758 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6523f25-d7c1-4881-8adc-788c8226ea9c" path="/var/lib/kubelet/pods/a6523f25-d7c1-4881-8adc-788c8226ea9c/volumes" Nov 02 01:55:20.366738 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:20.366700 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:20.515255 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:20.515217 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:21.046214 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:21.046177 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:21.305743 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:21.305612 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:24.145294 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:24.145034 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:24.189935 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:55:24.189894 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:55:24.634848 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:24.634807 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:25.362878 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.362839 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:25.522767 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.522733 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:25.553991 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.553961 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r"] Nov 02 01:55:25.554724 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.554705 5044 cpu_manager.go:414] "containerMap: removing container" podUID="a6523f25-d7c1-4881-8adc-788c8226ea9c" containerName="istio-proxy" Nov 02 01:55:25.554724 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.554724 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6523f25-d7c1-4881-8adc-788c8226ea9c" containerName="istio-proxy" Nov 02 01:55:25.554884 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.554874 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6523f25-d7c1-4881-8adc-788c8226ea9c" containerName="istio-proxy" Nov 02 01:55:25.558610 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.558587 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" Nov 02 01:55:25.652782 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.652655 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-a7c06d-serving-cert\"" Nov 02 01:55:25.658370 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.658334 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r"] Nov 02 01:55:25.671143 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.671114 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:25.675540 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.675517 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r\" (UID: \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" Nov 02 01:55:25.675630 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.675549 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r\" (UID: \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" Nov 02 01:55:25.675719 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.675699 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqn7v\" (UniqueName: \"kubernetes.io/projected/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-kube-api-access-fqn7v\") pod \"maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r\" (UID: \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" Nov 02 01:55:25.675826 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.675809 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r\" (UID: \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" Nov 02 01:55:25.781125 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.781083 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r\" (UID: \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" Nov 02 01:55:25.781610 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.781454 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r\" (UID: \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" Nov 02 01:55:25.781929 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.781911 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqn7v\" (UniqueName: \"kubernetes.io/projected/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-kube-api-access-fqn7v\") pod \"maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r\" (UID: \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" Nov 02 01:55:25.782272 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.782255 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r\" (UID: \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" Nov 02 01:55:25.786861 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:55:25.786837 5044 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-serving-cert: secret "maas-api-e2e-shared-a-w6-a7c06d-serving-cert" not found Nov 02 01:55:25.787017 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:55:25.786919 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-maas-api-tls podName:9bcbb6c9-810a-475f-915c-5c05b23a2f3f nodeName:}" failed. No retries permitted until 2026-11-02 01:55:26.286898075 +0000 UTC m=+1842.958150737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-maas-api-tls") pod "maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" (UID: "9bcbb6c9-810a-475f-915c-5c05b23a2f3f") : secret "maas-api-e2e-shared-a-w6-a7c06d-serving-cert" not found Nov 02 01:55:25.788137 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.788112 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r\" (UID: \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" Nov 02 01:55:25.804727 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.801891 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r\" (UID: \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" Nov 02 01:55:25.838695 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.825182 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:25.896694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.895385 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqn7v\" (UniqueName: \"kubernetes.io/projected/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-kube-api-access-fqn7v\") pod \"maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r\" (UID: \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" Nov 02 01:55:25.996077 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:25.995969 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:26.212081 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:26.212050 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:26.297608 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:26.297512 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r\" (UID: \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" Nov 02 01:55:26.300079 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:26.300054 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r\" (UID: \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" Nov 02 01:55:26.473323 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:26.473288 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" Nov 02 01:55:26.884576 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:26.884533 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r"] Nov 02 01:55:26.886060 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:55:26.886029 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bcbb6c9_810a_475f_915c_5c05b23a2f3f.slice/crio-fb000cf2aba01c7b6f9f124c880b68c8bd1bae990316c5613fa75c787795ef34 WatchSource:0}: Error finding container fb000cf2aba01c7b6f9f124c880b68c8bd1bae990316c5613fa75c787795ef34: Status 404 returned error can't find the container with id fb000cf2aba01c7b6f9f124c880b68c8bd1bae990316c5613fa75c787795ef34 Nov 02 01:55:27.537083 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:27.537046 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" event={"ID":"9bcbb6c9-810a-475f-915c-5c05b23a2f3f","Type":"ContainerStarted","Data":"fb000cf2aba01c7b6f9f124c880b68c8bd1bae990316c5613fa75c787795ef34"} Nov 02 01:55:28.545895 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:28.545856 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" event={"ID":"9bcbb6c9-810a-475f-915c-5c05b23a2f3f","Type":"ContainerStarted","Data":"bea8318e7b4a65de8282680e42952d3103555b6755f0b276a310ef759a8acb25"} Nov 02 01:55:28.546347 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:28.545937 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" Nov 02 01:55:28.667876 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:28.667823 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" podStartSLOduration=2.821628604 podStartE2EDuration="3.667805453s" totalImagesPullingTime="846.176849ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:55:28.665590361 +0000 UTC m=+1845.336843043" watchObservedRunningTime="2026-11-02 01:55:28.667805453 +0000 UTC m=+1845.339058135" Nov 02 01:55:29.824839 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:29.824806 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb"] Nov 02 01:55:29.825338 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:29.825014 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" podUID="07be5104-2adc-4dff-87bb-f115473caa5d" containerName="istio-proxy" containerID="cri-o://89054247c845b795c849c3db9c9213c741e423afce3a4f04bf7aacd55c2cf363" gracePeriod=30 Nov 02 01:55:30.361775 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:30.361734 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:30.525248 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:30.525210 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:30.675605 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:30.675515 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:31.026618 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:31.026584 5044 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:55:31.027092 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:31.026664 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" podUID="07be5104-2adc-4dff-87bb-f115473caa5d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:55:31.314215 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:31.314102 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:31.945346 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:31.945291 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:32.162121 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:32.162083 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:34.559594 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:34.559564 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" Nov 02 01:55:35.112957 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.112930 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:55:35.274219 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.274182 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07be5104-2adc-4dff-87bb-f115473caa5d-istio-token\" (UniqueName: \"kubernetes.io/projected/07be5104-2adc-4dff-87bb-f115473caa5d-istio-token\") pod \"07be5104-2adc-4dff-87bb-f115473caa5d\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " Nov 02 01:55:35.274415 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.274259 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07be5104-2adc-4dff-87bb-f115473caa5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07be5104-2adc-4dff-87bb-f115473caa5d-istiod-ca-cert\") pod \"07be5104-2adc-4dff-87bb-f115473caa5d\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " Nov 02 01:55:35.274415 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.274338 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-credential-socket\") pod \"07be5104-2adc-4dff-87bb-f115473caa5d\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " Nov 02 01:55:35.274415 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.274385 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-istio-envoy\") pod \"07be5104-2adc-4dff-87bb-f115473caa5d\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " Nov 02 01:55:35.274645 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.274513 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07be5104-2adc-4dff-87bb-f115473caa5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07be5104-2adc-4dff-87bb-f115473caa5d-istio-podinfo\") pod \"07be5104-2adc-4dff-87bb-f115473caa5d\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " Nov 02 01:55:35.274645 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.274564 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-workload-certs\") pod \"07be5104-2adc-4dff-87bb-f115473caa5d\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " Nov 02 01:55:35.274645 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.274594 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-workload-socket\") pod \"07be5104-2adc-4dff-87bb-f115473caa5d\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " Nov 02 01:55:35.274852 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.274646 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07be5104-2adc-4dff-87bb-f115473caa5d-kube-api-access-xb4zl\" (UniqueName: \"kubernetes.io/projected/07be5104-2adc-4dff-87bb-f115473caa5d-kube-api-access-xb4zl\") pod \"07be5104-2adc-4dff-87bb-f115473caa5d\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " Nov 02 01:55:35.277793 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.275043 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-istio-data\") pod \"07be5104-2adc-4dff-87bb-f115473caa5d\" (UID: \"07be5104-2adc-4dff-87bb-f115473caa5d\") " Nov 02 01:55:35.277793 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.275147 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-credential-socket" pod "07be5104-2adc-4dff-87bb-f115473caa5d" (UID: "07be5104-2adc-4dff-87bb-f115473caa5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:55:35.277793 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.275365 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-workload-certs" pod "07be5104-2adc-4dff-87bb-f115473caa5d" (UID: "07be5104-2adc-4dff-87bb-f115473caa5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:55:35.277793 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.275360 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07be5104-2adc-4dff-87bb-f115473caa5d-istiod-ca-cert" pod "07be5104-2adc-4dff-87bb-f115473caa5d" (UID: "07be5104-2adc-4dff-87bb-f115473caa5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:55:35.277793 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.276016 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-istio-data" pod "07be5104-2adc-4dff-87bb-f115473caa5d" (UID: "07be5104-2adc-4dff-87bb-f115473caa5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:55:35.277793 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.275790 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-workload-socket" pod "07be5104-2adc-4dff-87bb-f115473caa5d" (UID: "07be5104-2adc-4dff-87bb-f115473caa5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:55:35.277793 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.277152 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:55:35.277793 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.277177 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:55:35.277793 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.277191 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:55:35.277793 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.277203 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07be5104-2adc-4dff-87bb-f115473caa5d-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:55:35.277793 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.277217 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:55:35.282029 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.278943 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-istio-envoy" pod "07be5104-2adc-4dff-87bb-f115473caa5d" (UID: "07be5104-2adc-4dff-87bb-f115473caa5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:55:35.282029 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.280569 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07be5104-2adc-4dff-87bb-f115473caa5d-istio-podinfo" pod "07be5104-2adc-4dff-87bb-f115473caa5d" (UID: "07be5104-2adc-4dff-87bb-f115473caa5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:55:35.282029 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.281023 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07be5104-2adc-4dff-87bb-f115473caa5d-kube-api-access-xb4zl" pod "07be5104-2adc-4dff-87bb-f115473caa5d" (UID: "07be5104-2adc-4dff-87bb-f115473caa5d"). InnerVolumeSpecName "kube-api-access-xb4zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:55:35.282129 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.282079 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07be5104-2adc-4dff-87bb-f115473caa5d-istio-token" pod "07be5104-2adc-4dff-87bb-f115473caa5d" (UID: "07be5104-2adc-4dff-87bb-f115473caa5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:55:35.378514 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.378412 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07be5104-2adc-4dff-87bb-f115473caa5d-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:55:35.378514 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.378440 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07be5104-2adc-4dff-87bb-f115473caa5d-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:55:35.378514 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.378450 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb4zl\" (UniqueName: \"kubernetes.io/projected/07be5104-2adc-4dff-87bb-f115473caa5d-kube-api-access-xb4zl\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:55:35.378514 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.378461 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07be5104-2adc-4dff-87bb-f115473caa5d-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:55:35.624608 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.624570 5044 generic.go:356] "Generic (PLEG): container finished" podID="07be5104-2adc-4dff-87bb-f115473caa5d" containerID="89054247c845b795c849c3db9c9213c741e423afce3a4f04bf7aacd55c2cf363" exitCode=0 Nov 02 01:55:35.625103 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.624625 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" Nov 02 01:55:35.625103 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.624662 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" event={"ID":"07be5104-2adc-4dff-87bb-f115473caa5d","Type":"ContainerDied","Data":"89054247c845b795c849c3db9c9213c741e423afce3a4f04bf7aacd55c2cf363"} Nov 02 01:55:35.625103 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.624751 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb" event={"ID":"07be5104-2adc-4dff-87bb-f115473caa5d","Type":"ContainerDied","Data":"b0a7c6e64be5dcc46dea523b88cb19aca6f745517744a61eee39f402faaae8ff"} Nov 02 01:55:35.625103 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.624786 5044 scope.go:122] "RemoveContainer" containerID="89054247c845b795c849c3db9c9213c741e423afce3a4f04bf7aacd55c2cf363" Nov 02 01:55:35.635453 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.635433 5044 scope.go:122] "RemoveContainer" containerID="89054247c845b795c849c3db9c9213c741e423afce3a4f04bf7aacd55c2cf363" Nov 02 01:55:35.635805 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:55:35.635784 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89054247c845b795c849c3db9c9213c741e423afce3a4f04bf7aacd55c2cf363\": container with ID starting with 89054247c845b795c849c3db9c9213c741e423afce3a4f04bf7aacd55c2cf363 not found: ID does not exist" containerID="89054247c845b795c849c3db9c9213c741e423afce3a4f04bf7aacd55c2cf363" Nov 02 01:55:35.635878 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.635814 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89054247c845b795c849c3db9c9213c741e423afce3a4f04bf7aacd55c2cf363"} err="failed to get container status \"89054247c845b795c849c3db9c9213c741e423afce3a4f04bf7aacd55c2cf363\": rpc error: code = NotFound desc = could not find container \"89054247c845b795c849c3db9c9213c741e423afce3a4f04bf7aacd55c2cf363\": container with ID starting with 89054247c845b795c849c3db9c9213c741e423afce3a4f04bf7aacd55c2cf363 not found: ID does not exist" Nov 02 01:55:35.794079 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.794039 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb"] Nov 02 01:55:35.844707 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:35.844646 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-ztpvb"] Nov 02 01:55:36.629116 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:36.629080 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:37.458469 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:37.458429 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07be5104-2adc-4dff-87bb-f115473caa5d" path="/var/lib/kubelet/pods/07be5104-2adc-4dff-87bb-f115473caa5d/volumes" Nov 02 01:55:41.849753 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:41.849723 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:43.690982 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:43.690931 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:46.989947 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:46.989912 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:47.345632 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:47.345600 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:51.881112 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:51.881069 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:54.039435 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:54.039402 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:54.175416 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:55:54.175381 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:55:57.002940 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:57.002907 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:55:58.392721 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:55:58.392657 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:02.386126 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:02.386085 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:03.112275 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:03.112237 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:07.620147 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:07.620118 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:11.063697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:11.063648 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:11.479050 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:11.478975 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:12.109183 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:12.109148 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:12.516450 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:12.516414 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:13.173275 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:13.173242 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:17.241234 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:17.241194 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:17.410625 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:17.410587 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:17.562497 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:17.562465 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:18.337481 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:18.337442 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:19.864801 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:19.864761 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:20.021305 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:20.021272 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:24.217390 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:56:24.217352 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:56:24.223696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:24.220529 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:28.950407 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:28.950364 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd"] Nov 02 01:56:28.950909 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:28.950621 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" podUID="03580e97-7ea6-4a50-a1dd-640da600885c" containerName="istio-proxy" containerID="cri-o://c83336d6848b9edb5cc40289869fdf5a120713fa0e701a9e75f90bc82364f8f4" gracePeriod=30 Nov 02 01:56:29.316802 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:29.316768 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:32.148106 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:32.148074 5044 generic.go:356] "Generic (PLEG): container finished" podID="1098cb6c-8b36-4df8-87d9-357d3ed510e0" containerID="f9a7355bb7e05b8ad35aaeaf18aab7ec02f1ca3c2d163e0837e5d253a6305359" exitCode=0 Nov 02 01:56:32.148527 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:32.148147 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" event={"ID":"1098cb6c-8b36-4df8-87d9-357d3ed510e0","Type":"ContainerDied","Data":"f9a7355bb7e05b8ad35aaeaf18aab7ec02f1ca3c2d163e0837e5d253a6305359"} Nov 02 01:56:32.148586 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:32.148572 5044 scope.go:122] "RemoveContainer" containerID="f9a7355bb7e05b8ad35aaeaf18aab7ec02f1ca3c2d163e0837e5d253a6305359" Nov 02 01:56:32.150312 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:32.150287 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:56:32.150400 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:32.150352 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:56:32.150400 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:32.150388 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 01:56:32.993258 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:32.993208 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:56:32.993445 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:32.993292 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" podUID="03580e97-7ea6-4a50-a1dd-640da600885c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:56:33.158265 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:33.158221 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" event={"ID":"1098cb6c-8b36-4df8-87d9-357d3ed510e0","Type":"ContainerStarted","Data":"8d0c81b411e50d6058a15e53a285c2760aa057e4b0ecee478d611e03a50cd792"} Nov 02 01:56:33.432777 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:33.432744 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:56:33.432955 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:33.432917 5044 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 02 01:56:33.433000 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:33.432962 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" podUID="1098cb6c-8b36-4df8-87d9-357d3ed510e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 02 01:56:34.230410 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.230332 5044 generic.go:356] "Generic (PLEG): container finished" podID="03580e97-7ea6-4a50-a1dd-640da600885c" containerID="c83336d6848b9edb5cc40289869fdf5a120713fa0e701a9e75f90bc82364f8f4" exitCode=0 Nov 02 01:56:34.231042 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.230977 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" event={"ID":"03580e97-7ea6-4a50-a1dd-640da600885c","Type":"ContainerDied","Data":"c83336d6848b9edb5cc40289869fdf5a120713fa0e701a9e75f90bc82364f8f4"} Nov 02 01:56:34.240829 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.240799 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd"] Nov 02 01:56:34.241856 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.241837 5044 cpu_manager.go:414] "containerMap: removing container" podUID="07be5104-2adc-4dff-87bb-f115473caa5d" containerName="istio-proxy" Nov 02 01:56:34.241913 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.241859 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07be5104-2adc-4dff-87bb-f115473caa5d" containerName="istio-proxy" Nov 02 01:56:34.242101 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.242086 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="07be5104-2adc-4dff-87bb-f115473caa5d" containerName="istio-proxy" Nov 02 01:56:34.246213 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.246192 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" Nov 02 01:56:34.303627 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.303594 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-218802-serving-cert\"" Nov 02 01:56:34.308272 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.308245 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd"] Nov 02 01:56:34.354379 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.354351 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:56:34.418155 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.418122 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68f185a-890d-445d-9e1e-b35d7d0e2448-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd\" (UID: \"e68f185a-890d-445d-9e1e-b35d7d0e2448\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" Nov 02 01:56:34.418360 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.418255 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ql9n\" (UniqueName: \"kubernetes.io/projected/e68f185a-890d-445d-9e1e-b35d7d0e2448-kube-api-access-9ql9n\") pod \"maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd\" (UID: \"e68f185a-890d-445d-9e1e-b35d7d0e2448\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" Nov 02 01:56:34.418360 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.418332 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68f185a-890d-445d-9e1e-b35d7d0e2448-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd\" (UID: \"e68f185a-890d-445d-9e1e-b35d7d0e2448\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" Nov 02 01:56:34.418565 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.418543 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68f185a-890d-445d-9e1e-b35d7d0e2448-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd\" (UID: \"e68f185a-890d-445d-9e1e-b35d7d0e2448\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" Nov 02 01:56:34.433090 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.433053 5044 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 02 01:56:34.433218 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.433114 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" podUID="1098cb6c-8b36-4df8-87d9-357d3ed510e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 02 01:56:34.448426 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.448397 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:34.520382 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.520350 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-istio-data\") pod \"03580e97-7ea6-4a50-a1dd-640da600885c\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " Nov 02 01:56:34.520700 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.520660 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03580e97-7ea6-4a50-a1dd-640da600885c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03580e97-7ea6-4a50-a1dd-640da600885c-istiod-ca-cert\") pod \"03580e97-7ea6-4a50-a1dd-640da600885c\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " Nov 02 01:56:34.520977 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.520955 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-istio-envoy\") pod \"03580e97-7ea6-4a50-a1dd-640da600885c\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " Nov 02 01:56:34.521224 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.521204 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-credential-socket\") pod \"03580e97-7ea6-4a50-a1dd-640da600885c\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " Nov 02 01:56:34.521353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.521336 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03580e97-7ea6-4a50-a1dd-640da600885c-istio-token\" (UniqueName: \"kubernetes.io/projected/03580e97-7ea6-4a50-a1dd-640da600885c-istio-token\") pod \"03580e97-7ea6-4a50-a1dd-640da600885c\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " Nov 02 01:56:34.521489 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.521471 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-workload-socket\") pod \"03580e97-7ea6-4a50-a1dd-640da600885c\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " Nov 02 01:56:34.521644 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.521627 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03580e97-7ea6-4a50-a1dd-640da600885c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03580e97-7ea6-4a50-a1dd-640da600885c-istio-podinfo\") pod \"03580e97-7ea6-4a50-a1dd-640da600885c\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " Nov 02 01:56:34.521898 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.521878 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03580e97-7ea6-4a50-a1dd-640da600885c-kube-api-access-l9cs6\" (UniqueName: \"kubernetes.io/projected/03580e97-7ea6-4a50-a1dd-640da600885c-kube-api-access-l9cs6\") pod \"03580e97-7ea6-4a50-a1dd-640da600885c\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " Nov 02 01:56:34.531305 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.522106 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-workload-certs\") pod \"03580e97-7ea6-4a50-a1dd-640da600885c\" (UID: \"03580e97-7ea6-4a50-a1dd-640da600885c\") " Nov 02 01:56:34.532033 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.532006 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68f185a-890d-445d-9e1e-b35d7d0e2448-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd\" (UID: \"e68f185a-890d-445d-9e1e-b35d7d0e2448\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" Nov 02 01:56:34.610463 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.605289 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68f185a-890d-445d-9e1e-b35d7d0e2448-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd\" (UID: \"e68f185a-890d-445d-9e1e-b35d7d0e2448\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" Nov 02 01:56:34.610463 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.605476 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ql9n\" (UniqueName: \"kubernetes.io/projected/e68f185a-890d-445d-9e1e-b35d7d0e2448-kube-api-access-9ql9n\") pod \"maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd\" (UID: \"e68f185a-890d-445d-9e1e-b35d7d0e2448\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" Nov 02 01:56:34.610463 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.605510 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68f185a-890d-445d-9e1e-b35d7d0e2448-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd\" (UID: \"e68f185a-890d-445d-9e1e-b35d7d0e2448\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" Nov 02 01:56:34.610463 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.525441 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-credential-socket" pod "03580e97-7ea6-4a50-a1dd-640da600885c" (UID: "03580e97-7ea6-4a50-a1dd-640da600885c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:56:34.610463 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.525605 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-istio-data" pod "03580e97-7ea6-4a50-a1dd-640da600885c" (UID: "03580e97-7ea6-4a50-a1dd-640da600885c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:56:34.610463 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.525931 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03580e97-7ea6-4a50-a1dd-640da600885c-istiod-ca-cert" pod "03580e97-7ea6-4a50-a1dd-640da600885c" (UID: "03580e97-7ea6-4a50-a1dd-640da600885c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:56:34.610463 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.531985 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-workload-certs" pod "03580e97-7ea6-4a50-a1dd-640da600885c" (UID: "03580e97-7ea6-4a50-a1dd-640da600885c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:56:34.610463 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.550137 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03580e97-7ea6-4a50-a1dd-640da600885c-kube-api-access-l9cs6" pod "03580e97-7ea6-4a50-a1dd-640da600885c" (UID: "03580e97-7ea6-4a50-a1dd-640da600885c"). InnerVolumeSpecName "kube-api-access-l9cs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:56:34.610463 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.550537 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-workload-socket" pod "03580e97-7ea6-4a50-a1dd-640da600885c" (UID: "03580e97-7ea6-4a50-a1dd-640da600885c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:56:34.610463 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.593832 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03580e97-7ea6-4a50-a1dd-640da600885c-istio-podinfo" pod "03580e97-7ea6-4a50-a1dd-640da600885c" (UID: "03580e97-7ea6-4a50-a1dd-640da600885c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:56:34.610463 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:56:34.605732 5044 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-serving-cert: secret "maas-api-e2e-shared-a-w3-218802-serving-cert" not found Nov 02 01:56:34.610463 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:56:34.605861 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68f185a-890d-445d-9e1e-b35d7d0e2448-maas-api-tls podName:e68f185a-890d-445d-9e1e-b35d7d0e2448 nodeName:}" failed. No retries permitted until 2026-11-02 01:56:35.10583749 +0000 UTC m=+1911.777090154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e68f185a-890d-445d-9e1e-b35d7d0e2448-maas-api-tls") pod "maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" (UID: "e68f185a-890d-445d-9e1e-b35d7d0e2448") : secret "maas-api-e2e-shared-a-w3-218802-serving-cert" not found Nov 02 01:56:34.610463 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.606837 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68f185a-890d-445d-9e1e-b35d7d0e2448-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd\" (UID: \"e68f185a-890d-445d-9e1e-b35d7d0e2448\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" Nov 02 01:56:34.610463 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.544491 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68f185a-890d-445d-9e1e-b35d7d0e2448-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd\" (UID: \"e68f185a-890d-445d-9e1e-b35d7d0e2448\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" Nov 02 01:56:34.610463 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.609212 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:34.612085 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.612056 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-istio-envoy" pod "03580e97-7ea6-4a50-a1dd-640da600885c" (UID: "03580e97-7ea6-4a50-a1dd-640da600885c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:56:34.614230 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.614204 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03580e97-7ea6-4a50-a1dd-640da600885c-istio-token" pod "03580e97-7ea6-4a50-a1dd-640da600885c" (UID: "03580e97-7ea6-4a50-a1dd-640da600885c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:56:34.698444 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.698410 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ql9n\" (UniqueName: \"kubernetes.io/projected/e68f185a-890d-445d-9e1e-b35d7d0e2448-kube-api-access-9ql9n\") pod \"maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd\" (UID: \"e68f185a-890d-445d-9e1e-b35d7d0e2448\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" Nov 02 01:56:34.706819 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.706792 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:56:34.706819 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.706816 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:56:34.706995 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.706827 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03580e97-7ea6-4a50-a1dd-640da600885c-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:56:34.706995 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.706836 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:56:34.706995 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.706844 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03580e97-7ea6-4a50-a1dd-640da600885c-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:56:34.706995 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.706854 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9cs6\" (UniqueName: \"kubernetes.io/projected/03580e97-7ea6-4a50-a1dd-640da600885c-kube-api-access-l9cs6\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:56:34.706995 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.706864 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:56:34.706995 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.706872 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03580e97-7ea6-4a50-a1dd-640da600885c-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:56:34.706995 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.706880 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03580e97-7ea6-4a50-a1dd-640da600885c-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:56:34.820794 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:34.820700 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:35.115289 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:35.115204 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68f185a-890d-445d-9e1e-b35d7d0e2448-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd\" (UID: \"e68f185a-890d-445d-9e1e-b35d7d0e2448\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" Nov 02 01:56:35.117724 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:35.117698 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68f185a-890d-445d-9e1e-b35d7d0e2448-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd\" (UID: \"e68f185a-890d-445d-9e1e-b35d7d0e2448\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" Nov 02 01:56:35.181694 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:35.179307 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" Nov 02 01:56:35.240087 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:35.240062 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" Nov 02 01:56:35.240087 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:35.240069 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd" event={"ID":"03580e97-7ea6-4a50-a1dd-640da600885c","Type":"ContainerDied","Data":"bdb890105b15fefb8a8c69ffe27cfb7622f0caf841e6b104ef107874be33d4a1"} Nov 02 01:56:35.240511 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:35.240169 5044 scope.go:122] "RemoveContainer" containerID="c83336d6848b9edb5cc40289869fdf5a120713fa0e701a9e75f90bc82364f8f4" Nov 02 01:56:35.412445 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:35.412366 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd"] Nov 02 01:56:35.433705 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:35.433656 5044 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 02 01:56:35.433885 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:35.433737 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" podUID="1098cb6c-8b36-4df8-87d9-357d3ed510e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 02 01:56:35.482703 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:35.464761 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-vsmqd"] Nov 02 01:56:35.590543 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:35.590505 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd"] Nov 02 01:56:35.594172 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:56:35.594142 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68f185a_890d_445d_9e1e_b35d7d0e2448.slice/crio-179fca984674e4790b7948285834751e9e0634fe4a4ab6bc8a10918893aa8daa WatchSource:0}: Error finding container 179fca984674e4790b7948285834751e9e0634fe4a4ab6bc8a10918893aa8daa: Status 404 returned error can't find the container with id 179fca984674e4790b7948285834751e9e0634fe4a4ab6bc8a10918893aa8daa Nov 02 01:56:36.249609 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:36.249579 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" event={"ID":"e68f185a-890d-445d-9e1e-b35d7d0e2448","Type":"ContainerStarted","Data":"179fca984674e4790b7948285834751e9e0634fe4a4ab6bc8a10918893aa8daa"} Nov 02 01:56:36.433663 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:36.433626 5044 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 02 01:56:36.433878 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:36.433725 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" podUID="1098cb6c-8b36-4df8-87d9-357d3ed510e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 02 01:56:37.257902 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:37.257870 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" event={"ID":"e68f185a-890d-445d-9e1e-b35d7d0e2448","Type":"ContainerStarted","Data":"75b02b3181d3161f137f276f95923835a6a3bae0ebbb271424665214d3a80445"} Nov 02 01:56:37.258332 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:37.257957 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" Nov 02 01:56:37.421697 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:37.420806 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" podStartSLOduration=2.4861737760000002 podStartE2EDuration="3.420786799s" totalImagesPullingTime="934.613023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:56:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:56:37.382118477 +0000 UTC m=+1914.053371158" watchObservedRunningTime="2026-11-02 01:56:37.420786799 +0000 UTC m=+1914.092039481" Nov 02 01:56:37.437698 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:37.434105 5044 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 02 01:56:37.437698 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:37.434166 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" podUID="1098cb6c-8b36-4df8-87d9-357d3ed510e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 02 01:56:37.505794 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:37.505736 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03580e97-7ea6-4a50-a1dd-640da600885c" path="/var/lib/kubelet/pods/03580e97-7ea6-4a50-a1dd-640da600885c/volumes" Nov 02 01:56:38.432953 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:38.432911 5044 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 02 01:56:38.433376 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:38.432985 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" podUID="1098cb6c-8b36-4df8-87d9-357d3ed510e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 02 01:56:38.683789 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:38.683718 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:56:39.437837 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:39.437806 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:56:39.439107 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:39.439086 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 01:56:39.629038 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:39.629001 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:40.314561 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:40.314524 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:40.527827 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:40.527789 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:41.863394 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:41.863355 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:43.296194 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:43.296166 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" Nov 02 01:56:44.082557 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.082521 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:44.451778 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.451697 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp"] Nov 02 01:56:44.452607 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.452587 5044 cpu_manager.go:414] "containerMap: removing container" podUID="03580e97-7ea6-4a50-a1dd-640da600885c" containerName="istio-proxy" Nov 02 01:56:44.452662 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.452609 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03580e97-7ea6-4a50-a1dd-640da600885c" containerName="istio-proxy" Nov 02 01:56:44.452786 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.452773 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="03580e97-7ea6-4a50-a1dd-640da600885c" containerName="istio-proxy" Nov 02 01:56:44.457594 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.457575 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" Nov 02 01:56:44.498442 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.498415 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46e40d47-ab73-4064-a9a2-f399cbd38c88-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp\" (UID: \"46e40d47-ab73-4064-a9a2-f399cbd38c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" Nov 02 01:56:44.498580 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.498455 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ndxr\" (UniqueName: \"kubernetes.io/projected/46e40d47-ab73-4064-a9a2-f399cbd38c88-kube-api-access-2ndxr\") pod \"maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp\" (UID: \"46e40d47-ab73-4064-a9a2-f399cbd38c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" Nov 02 01:56:44.498648 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.498610 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46e40d47-ab73-4064-a9a2-f399cbd38c88-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp\" (UID: \"46e40d47-ab73-4064-a9a2-f399cbd38c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" Nov 02 01:56:44.498833 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.498810 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e40d47-ab73-4064-a9a2-f399cbd38c88-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp\" (UID: \"46e40d47-ab73-4064-a9a2-f399cbd38c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" Nov 02 01:56:44.514415 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.514391 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp"] Nov 02 01:56:44.560798 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.560775 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-bb0464-serving-cert\"" Nov 02 01:56:44.601949 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.599650 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46e40d47-ab73-4064-a9a2-f399cbd38c88-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp\" (UID: \"46e40d47-ab73-4064-a9a2-f399cbd38c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" Nov 02 01:56:44.601949 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.600536 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ndxr\" (UniqueName: \"kubernetes.io/projected/46e40d47-ab73-4064-a9a2-f399cbd38c88-kube-api-access-2ndxr\") pod \"maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp\" (UID: \"46e40d47-ab73-4064-a9a2-f399cbd38c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" Nov 02 01:56:44.601949 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.600737 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46e40d47-ab73-4064-a9a2-f399cbd38c88-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp\" (UID: \"46e40d47-ab73-4064-a9a2-f399cbd38c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" Nov 02 01:56:44.601949 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.600882 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e40d47-ab73-4064-a9a2-f399cbd38c88-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp\" (UID: \"46e40d47-ab73-4064-a9a2-f399cbd38c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" Nov 02 01:56:44.601949 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.600455 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46e40d47-ab73-4064-a9a2-f399cbd38c88-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp\" (UID: \"46e40d47-ab73-4064-a9a2-f399cbd38c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" Nov 02 01:56:44.601949 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:56:44.601500 5044 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-serving-cert: secret "maas-api-e2e-shared-b-w4-bb0464-serving-cert" not found Nov 02 01:56:44.601949 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:56:44.601570 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46e40d47-ab73-4064-a9a2-f399cbd38c88-maas-api-tls podName:46e40d47-ab73-4064-a9a2-f399cbd38c88 nodeName:}" failed. No retries permitted until 2026-11-02 01:56:45.101553707 +0000 UTC m=+1921.772806370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/46e40d47-ab73-4064-a9a2-f399cbd38c88-maas-api-tls") pod "maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" (UID: "46e40d47-ab73-4064-a9a2-f399cbd38c88") : secret "maas-api-e2e-shared-b-w4-bb0464-serving-cert" not found Nov 02 01:56:44.604965 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.604903 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e40d47-ab73-4064-a9a2-f399cbd38c88-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp\" (UID: \"46e40d47-ab73-4064-a9a2-f399cbd38c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" Nov 02 01:56:44.637138 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.637112 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:44.665940 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.665912 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ndxr\" (UniqueName: \"kubernetes.io/projected/46e40d47-ab73-4064-a9a2-f399cbd38c88-kube-api-access-2ndxr\") pod \"maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp\" (UID: \"46e40d47-ab73-4064-a9a2-f399cbd38c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" Nov 02 01:56:44.798503 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.798472 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:44.944801 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:44.944770 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:45.108086 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:45.108002 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46e40d47-ab73-4064-a9a2-f399cbd38c88-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp\" (UID: \"46e40d47-ab73-4064-a9a2-f399cbd38c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" Nov 02 01:56:45.110490 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:45.110463 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46e40d47-ab73-4064-a9a2-f399cbd38c88-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp\" (UID: \"46e40d47-ab73-4064-a9a2-f399cbd38c88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" Nov 02 01:56:45.372343 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:45.372261 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" Nov 02 01:56:45.753825 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:56:45.753794 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e40d47_ab73_4064_a9a2_f399cbd38c88.slice/crio-672ab0c8ad411460b51e555ee7744a5e6c8d4353aae4745b1930a7613dfe1cda WatchSource:0}: Error finding container 672ab0c8ad411460b51e555ee7744a5e6c8d4353aae4745b1930a7613dfe1cda: Status 404 returned error can't find the container with id 672ab0c8ad411460b51e555ee7744a5e6c8d4353aae4745b1930a7613dfe1cda Nov 02 01:56:45.755090 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:45.755062 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp"] Nov 02 01:56:46.356434 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:46.356397 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" event={"ID":"46e40d47-ab73-4064-a9a2-f399cbd38c88","Type":"ContainerStarted","Data":"672ab0c8ad411460b51e555ee7744a5e6c8d4353aae4745b1930a7613dfe1cda"} Nov 02 01:56:47.365775 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:47.365740 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" event={"ID":"46e40d47-ab73-4064-a9a2-f399cbd38c88","Type":"ContainerStarted","Data":"f0eee60ce90122dfd97d94a3a495aad583cbefdab8586de5a2d16a613f5393ca"} Nov 02 01:56:47.366188 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:47.365954 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" Nov 02 01:56:47.488615 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:47.488567 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" podStartSLOduration=2.6208520699999998 podStartE2EDuration="3.488552926s" totalImagesPullingTime="867.700856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:56:47.486239979 +0000 UTC m=+1924.157492661" watchObservedRunningTime="2026-11-02 01:56:47.488552926 +0000 UTC m=+1924.159805606" Nov 02 01:56:48.987747 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:48.987711 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:49.306599 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:49.306561 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:49.460878 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:49.460844 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:50.223066 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:50.222998 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:50.413573 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:50.413502 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:50.564955 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:50.564921 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:50.739733 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:50.737251 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:51.137348 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:51.137315 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:51.464844 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:51.464759 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:51.759873 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:51.759844 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:52.153287 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:52.153207 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:53.380527 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:53.380452 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" Nov 02 01:56:54.177091 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:56:54.177055 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:56:54.824065 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:54.824037 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:55.025879 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:55.025848 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:55.177313 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:55.177237 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:57.038497 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:57.038467 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:59.003034 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:59.003006 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:59.150782 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:59.150705 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:56:59.353746 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:56:59.353713 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:02.256997 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:02.256957 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:04.213016 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:04.212977 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:05.007518 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:04.991603 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:05.418977 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:05.418923 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:07.863250 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:07.863192 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:09.680122 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:09.680089 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:09.899390 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:09.899350 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:10.054596 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:10.054564 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:13.232487 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:13.232455 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:13.759274 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:13.759242 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z"] Nov 02 01:57:13.759493 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:13.759473 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" podUID="c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" containerName="istio-proxy" containerID="cri-o://a9a76497fb4f0664d8b59400eaa47cb250978aaec2df521d33bf7a5ba12cb763" gracePeriod=30 Nov 02 01:57:14.133948 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:14.133858 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm"] Nov 02 01:57:14.134146 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:14.134117 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" podUID="e8266d54-e810-4f6d-8720-f903ee12ce08" containerName="istio-proxy" containerID="cri-o://fccaaeb93f7e4600725c391382c0d4b133386548512a049172f50f83f0ec66e5" gracePeriod=30 Nov 02 01:57:15.685903 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:15.685864 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74"] Nov 02 01:57:15.686419 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:15.686163 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" podUID="5952e0fe-3c89-4ec9-b8b0-79842c388b67" containerName="istio-proxy" containerID="cri-o://c70cba7777ad36b84fd21d150ee3dc9d823708a6fc55e180ed645028cca3e976" gracePeriod=30 Nov 02 01:57:17.513496 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:17.513461 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:17.705963 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:17.705930 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:17.777100 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:17.777005 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:57:17.777100 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:17.777084 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" podUID="c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:57:18.296661 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:18.296614 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:18.538885 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:18.538853 5044 patch_prober.go:29] interesting pod/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:57:18.539277 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:18.538913 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" podUID="5952e0fe-3c89-4ec9-b8b0-79842c388b67" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:57:18.709022 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:18.708931 5044 patch_prober.go:29] interesting pod/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:57:18.709022 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:18.708998 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" podUID="e8266d54-e810-4f6d-8720-f903ee12ce08" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:57:19.059094 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.059068 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:57:19.209690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.206850 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-workload-socket\") pod \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " Nov 02 01:57:19.209690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.206932 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-data\") pod \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " Nov 02 01:57:19.209690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.206959 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-credential-socket\") pod \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " Nov 02 01:57:19.209690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.207056 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istiod-ca-cert\") pod \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " Nov 02 01:57:19.209690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.207121 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-workload-socket" pod "c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" (UID: "c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:57:19.209690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.207157 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-workload-certs\") pod \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " Nov 02 01:57:19.209690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.207198 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-envoy\") pod \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " Nov 02 01:57:19.209690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.207298 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-kube-api-access-r4wjc\" (UniqueName: \"kubernetes.io/projected/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-kube-api-access-r4wjc\") pod \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " Nov 02 01:57:19.209690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.207302 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-data" pod "c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" (UID: "c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:57:19.209690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.207365 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-token\" (UniqueName: \"kubernetes.io/projected/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-token\") pod \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " Nov 02 01:57:19.209690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.207435 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-podinfo\") pod \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\" (UID: \"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e\") " Nov 02 01:57:19.209690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.207550 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istiod-ca-cert" pod "c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" (UID: "c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:57:19.209690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.207757 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-credential-socket" pod "c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" (UID: "c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:57:19.209690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.208008 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:19.209690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.208027 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:19.209690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.208042 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:19.209690 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.208057 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:19.210946 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.210919 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-workload-certs" pod "c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" (UID: "c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:57:19.212520 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.212495 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-token" pod "c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" (UID: "c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:57:19.212913 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.212859 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-podinfo" pod "c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" (UID: "c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:57:19.213561 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.213537 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-envoy" pod "c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" (UID: "c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:57:19.214793 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.214769 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-kube-api-access-r4wjc" pod "c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" (UID: "c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e"). InnerVolumeSpecName "kube-api-access-r4wjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:57:19.310882 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.309600 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:19.310882 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.309629 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:19.310882 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.309639 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4wjc\" (UniqueName: \"kubernetes.io/projected/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-kube-api-access-r4wjc\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:19.310882 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.309650 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:19.310882 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.309658 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:19.428042 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.428019 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:57:19.512532 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.512499 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-podinfo\") pod \"e8266d54-e810-4f6d-8720-f903ee12ce08\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " Nov 02 01:57:19.512713 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.512633 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-data\") pod \"e8266d54-e810-4f6d-8720-f903ee12ce08\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " Nov 02 01:57:19.512793 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.512747 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-workload-certs\") pod \"e8266d54-e810-4f6d-8720-f903ee12ce08\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " Nov 02 01:57:19.512875 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.512849 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8266d54-e810-4f6d-8720-f903ee12ce08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8266d54-e810-4f6d-8720-f903ee12ce08-istiod-ca-cert\") pod \"e8266d54-e810-4f6d-8720-f903ee12ce08\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " Nov 02 01:57:19.512996 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.512893 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-credential-socket\") pod \"e8266d54-e810-4f6d-8720-f903ee12ce08\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " Nov 02 01:57:19.512996 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.512923 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-token\" (UniqueName: \"kubernetes.io/projected/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-token\") pod \"e8266d54-e810-4f6d-8720-f903ee12ce08\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " Nov 02 01:57:19.512996 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.512942 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-workload-socket\") pod \"e8266d54-e810-4f6d-8720-f903ee12ce08\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " Nov 02 01:57:19.512996 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.512973 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-envoy\") pod \"e8266d54-e810-4f6d-8720-f903ee12ce08\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " Nov 02 01:57:19.513154 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.513023 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8266d54-e810-4f6d-8720-f903ee12ce08-kube-api-access-z2vk7\" (UniqueName: \"kubernetes.io/projected/e8266d54-e810-4f6d-8720-f903ee12ce08-kube-api-access-z2vk7\") pod \"e8266d54-e810-4f6d-8720-f903ee12ce08\" (UID: \"e8266d54-e810-4f6d-8720-f903ee12ce08\") " Nov 02 01:57:19.514043 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.514020 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8266d54-e810-4f6d-8720-f903ee12ce08-istiod-ca-cert" pod "e8266d54-e810-4f6d-8720-f903ee12ce08" (UID: "e8266d54-e810-4f6d-8720-f903ee12ce08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:57:19.514878 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.514853 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-data" pod "e8266d54-e810-4f6d-8720-f903ee12ce08" (UID: "e8266d54-e810-4f6d-8720-f903ee12ce08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:57:19.514970 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.514936 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-workload-certs" pod "e8266d54-e810-4f6d-8720-f903ee12ce08" (UID: "e8266d54-e810-4f6d-8720-f903ee12ce08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:57:19.515149 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.515129 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-credential-socket" pod "e8266d54-e810-4f6d-8720-f903ee12ce08" (UID: "e8266d54-e810-4f6d-8720-f903ee12ce08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:57:19.515342 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.515324 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-workload-socket" pod "e8266d54-e810-4f6d-8720-f903ee12ce08" (UID: "e8266d54-e810-4f6d-8720-f903ee12ce08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:57:19.515746 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.515710 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8266d54-e810-4f6d-8720-f903ee12ce08-kube-api-access-z2vk7" pod "e8266d54-e810-4f6d-8720-f903ee12ce08" (UID: "e8266d54-e810-4f6d-8720-f903ee12ce08"). InnerVolumeSpecName "kube-api-access-z2vk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:57:19.516859 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.516830 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-podinfo" pod "e8266d54-e810-4f6d-8720-f903ee12ce08" (UID: "e8266d54-e810-4f6d-8720-f903ee12ce08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:57:19.516962 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.516933 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-token" pod "e8266d54-e810-4f6d-8720-f903ee12ce08" (UID: "e8266d54-e810-4f6d-8720-f903ee12ce08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:57:19.517940 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.517918 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-envoy" pod "e8266d54-e810-4f6d-8720-f903ee12ce08" (UID: "e8266d54-e810-4f6d-8720-f903ee12ce08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:57:19.615323 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.615242 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:19.615323 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.615269 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:19.615323 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.615282 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:19.615323 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.615296 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:19.615323 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.615309 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2vk7\" (UniqueName: \"kubernetes.io/projected/e8266d54-e810-4f6d-8720-f903ee12ce08-kube-api-access-z2vk7\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:19.615323 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.615323 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:19.615884 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.615335 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:19.615884 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.615348 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8266d54-e810-4f6d-8720-f903ee12ce08-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:19.615884 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.615360 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8266d54-e810-4f6d-8720-f903ee12ce08-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:19.658149 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.658116 5044 generic.go:356] "Generic (PLEG): container finished" podID="e8266d54-e810-4f6d-8720-f903ee12ce08" containerID="fccaaeb93f7e4600725c391382c0d4b133386548512a049172f50f83f0ec66e5" exitCode=0 Nov 02 01:57:19.658305 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.658165 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" Nov 02 01:57:19.658305 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.658198 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" event={"ID":"e8266d54-e810-4f6d-8720-f903ee12ce08","Type":"ContainerDied","Data":"fccaaeb93f7e4600725c391382c0d4b133386548512a049172f50f83f0ec66e5"} Nov 02 01:57:19.658305 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.658268 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm" event={"ID":"e8266d54-e810-4f6d-8720-f903ee12ce08","Type":"ContainerDied","Data":"ad22594bd2c4b3a8e087c42dad4bc81eb1b2f19ea433924b9f13f52362517987"} Nov 02 01:57:19.658305 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.658298 5044 scope.go:122] "RemoveContainer" containerID="fccaaeb93f7e4600725c391382c0d4b133386548512a049172f50f83f0ec66e5" Nov 02 01:57:19.666901 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.662592 5044 generic.go:356] "Generic (PLEG): container finished" podID="c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" containerID="a9a76497fb4f0664d8b59400eaa47cb250978aaec2df521d33bf7a5ba12cb763" exitCode=0 Nov 02 01:57:19.666901 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.662814 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" event={"ID":"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e","Type":"ContainerDied","Data":"a9a76497fb4f0664d8b59400eaa47cb250978aaec2df521d33bf7a5ba12cb763"} Nov 02 01:57:19.666901 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.663246 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" Nov 02 01:57:19.666901 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.662873 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z" event={"ID":"c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e","Type":"ContainerDied","Data":"6cfb6cbd8ca353e13600f862826292f3d9b0e3373a4149ed50fc71a8622b5e47"} Nov 02 01:57:19.679057 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.679035 5044 scope.go:122] "RemoveContainer" containerID="fccaaeb93f7e4600725c391382c0d4b133386548512a049172f50f83f0ec66e5" Nov 02 01:57:19.679301 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:57:19.679280 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fccaaeb93f7e4600725c391382c0d4b133386548512a049172f50f83f0ec66e5\": container with ID starting with fccaaeb93f7e4600725c391382c0d4b133386548512a049172f50f83f0ec66e5 not found: ID does not exist" containerID="fccaaeb93f7e4600725c391382c0d4b133386548512a049172f50f83f0ec66e5" Nov 02 01:57:19.679375 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.679310 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fccaaeb93f7e4600725c391382c0d4b133386548512a049172f50f83f0ec66e5"} err="failed to get container status \"fccaaeb93f7e4600725c391382c0d4b133386548512a049172f50f83f0ec66e5\": rpc error: code = NotFound desc = could not find container \"fccaaeb93f7e4600725c391382c0d4b133386548512a049172f50f83f0ec66e5\": container with ID starting with fccaaeb93f7e4600725c391382c0d4b133386548512a049172f50f83f0ec66e5 not found: ID does not exist" Nov 02 01:57:19.679375 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.679328 5044 scope.go:122] "RemoveContainer" containerID="a9a76497fb4f0664d8b59400eaa47cb250978aaec2df521d33bf7a5ba12cb763" Nov 02 01:57:19.689089 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.689069 5044 scope.go:122] "RemoveContainer" containerID="a9a76497fb4f0664d8b59400eaa47cb250978aaec2df521d33bf7a5ba12cb763" Nov 02 01:57:19.689408 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:57:19.689383 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9a76497fb4f0664d8b59400eaa47cb250978aaec2df521d33bf7a5ba12cb763\": container with ID starting with a9a76497fb4f0664d8b59400eaa47cb250978aaec2df521d33bf7a5ba12cb763 not found: ID does not exist" containerID="a9a76497fb4f0664d8b59400eaa47cb250978aaec2df521d33bf7a5ba12cb763" Nov 02 01:57:19.689491 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.689420 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9a76497fb4f0664d8b59400eaa47cb250978aaec2df521d33bf7a5ba12cb763"} err="failed to get container status \"a9a76497fb4f0664d8b59400eaa47cb250978aaec2df521d33bf7a5ba12cb763\": rpc error: code = NotFound desc = could not find container \"a9a76497fb4f0664d8b59400eaa47cb250978aaec2df521d33bf7a5ba12cb763\": container with ID starting with a9a76497fb4f0664d8b59400eaa47cb250978aaec2df521d33bf7a5ba12cb763 not found: ID does not exist" Nov 02 01:57:19.834434 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.834401 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z"] Nov 02 01:57:19.837646 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.837623 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-rgb8z"] Nov 02 01:57:19.950081 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.950008 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm"] Nov 02 01:57:19.953787 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:19.953766 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-5pfrm"] Nov 02 01:57:21.024567 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.024530 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:57:21.044817 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.044783 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-data\") pod \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " Nov 02 01:57:21.044991 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.044893 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-token\" (UniqueName: \"kubernetes.io/projected/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-token\") pod \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " Nov 02 01:57:21.044991 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.044914 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5952e0fe-3c89-4ec9-b8b0-79842c388b67-kube-api-access-6zgk4\" (UniqueName: \"kubernetes.io/projected/5952e0fe-3c89-4ec9-b8b0-79842c388b67-kube-api-access-6zgk4\") pod \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " Nov 02 01:57:21.044991 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.044953 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-envoy\") pod \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " Nov 02 01:57:21.045170 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.044988 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-data" pod "5952e0fe-3c89-4ec9-b8b0-79842c388b67" (UID: "5952e0fe-3c89-4ec9-b8b0-79842c388b67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:57:21.045170 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.045029 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-credential-socket\") pod \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " Nov 02 01:57:21.045170 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.045075 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-podinfo\") pod \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " Nov 02 01:57:21.045170 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.045108 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-workload-certs\") pod \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " Nov 02 01:57:21.045375 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.045211 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istiod-ca-cert\") pod \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " Nov 02 01:57:21.045375 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.045300 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-workload-socket\") pod \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\" (UID: \"5952e0fe-3c89-4ec9-b8b0-79842c388b67\") " Nov 02 01:57:21.046060 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.045298 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-credential-socket" pod "5952e0fe-3c89-4ec9-b8b0-79842c388b67" (UID: "5952e0fe-3c89-4ec9-b8b0-79842c388b67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:57:21.046060 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.045702 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-workload-socket" pod "5952e0fe-3c89-4ec9-b8b0-79842c388b67" (UID: "5952e0fe-3c89-4ec9-b8b0-79842c388b67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:57:21.046060 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.045611 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istiod-ca-cert" pod "5952e0fe-3c89-4ec9-b8b0-79842c388b67" (UID: "5952e0fe-3c89-4ec9-b8b0-79842c388b67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:57:21.046060 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.045872 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-workload-certs" pod "5952e0fe-3c89-4ec9-b8b0-79842c388b67" (UID: "5952e0fe-3c89-4ec9-b8b0-79842c388b67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:57:21.046373 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.046080 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:21.046373 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.046100 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:21.046373 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.046114 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:21.046373 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.046128 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:21.046373 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.046141 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:21.047379 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.047353 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-podinfo" pod "5952e0fe-3c89-4ec9-b8b0-79842c388b67" (UID: "5952e0fe-3c89-4ec9-b8b0-79842c388b67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:57:21.047520 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.047497 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-token" pod "5952e0fe-3c89-4ec9-b8b0-79842c388b67" (UID: "5952e0fe-3c89-4ec9-b8b0-79842c388b67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:57:21.047635 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.047564 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-envoy" pod "5952e0fe-3c89-4ec9-b8b0-79842c388b67" (UID: "5952e0fe-3c89-4ec9-b8b0-79842c388b67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:57:21.047635 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.047580 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5952e0fe-3c89-4ec9-b8b0-79842c388b67-kube-api-access-6zgk4" pod "5952e0fe-3c89-4ec9-b8b0-79842c388b67" (UID: "5952e0fe-3c89-4ec9-b8b0-79842c388b67"). InnerVolumeSpecName "kube-api-access-6zgk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:57:21.147686 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.147640 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:21.147686 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.147688 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zgk4\" (UniqueName: \"kubernetes.io/projected/5952e0fe-3c89-4ec9-b8b0-79842c388b67-kube-api-access-6zgk4\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:21.147880 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.147698 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:21.147880 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.147707 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5952e0fe-3c89-4ec9-b8b0-79842c388b67-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:57:21.453855 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.453821 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" path="/var/lib/kubelet/pods/c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e/volumes" Nov 02 01:57:21.454357 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.454340 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8266d54-e810-4f6d-8720-f903ee12ce08" path="/var/lib/kubelet/pods/e8266d54-e810-4f6d-8720-f903ee12ce08/volumes" Nov 02 01:57:21.682431 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.682401 5044 generic.go:356] "Generic (PLEG): container finished" podID="5952e0fe-3c89-4ec9-b8b0-79842c388b67" containerID="c70cba7777ad36b84fd21d150ee3dc9d823708a6fc55e180ed645028cca3e976" exitCode=0 Nov 02 01:57:21.682614 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.682461 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" Nov 02 01:57:21.682614 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.682477 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" event={"ID":"5952e0fe-3c89-4ec9-b8b0-79842c388b67","Type":"ContainerDied","Data":"c70cba7777ad36b84fd21d150ee3dc9d823708a6fc55e180ed645028cca3e976"} Nov 02 01:57:21.682614 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.682547 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74" event={"ID":"5952e0fe-3c89-4ec9-b8b0-79842c388b67","Type":"ContainerDied","Data":"8b47e6a8fd5dd9b988d13568441f779e970e4ca57c61ad75bc99ceed2ac062dd"} Nov 02 01:57:21.682614 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.682574 5044 scope.go:122] "RemoveContainer" containerID="c70cba7777ad36b84fd21d150ee3dc9d823708a6fc55e180ed645028cca3e976" Nov 02 01:57:21.704141 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.704082 5044 scope.go:122] "RemoveContainer" containerID="c70cba7777ad36b84fd21d150ee3dc9d823708a6fc55e180ed645028cca3e976" Nov 02 01:57:21.704736 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:57:21.704703 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c70cba7777ad36b84fd21d150ee3dc9d823708a6fc55e180ed645028cca3e976\": container with ID starting with c70cba7777ad36b84fd21d150ee3dc9d823708a6fc55e180ed645028cca3e976 not found: ID does not exist" containerID="c70cba7777ad36b84fd21d150ee3dc9d823708a6fc55e180ed645028cca3e976" Nov 02 01:57:21.704873 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.704746 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c70cba7777ad36b84fd21d150ee3dc9d823708a6fc55e180ed645028cca3e976"} err="failed to get container status \"c70cba7777ad36b84fd21d150ee3dc9d823708a6fc55e180ed645028cca3e976\": rpc error: code = NotFound desc = could not find container \"c70cba7777ad36b84fd21d150ee3dc9d823708a6fc55e180ed645028cca3e976\": container with ID starting with c70cba7777ad36b84fd21d150ee3dc9d823708a6fc55e180ed645028cca3e976 not found: ID does not exist" Nov 02 01:57:21.853919 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.853882 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74"] Nov 02 01:57:21.904608 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:21.904574 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-36a1e0-openshift-default-598d485c-8mb74"] Nov 02 01:57:23.392145 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:23.392082 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:23.457902 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:23.457870 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5952e0fe-3c89-4ec9-b8b0-79842c388b67" path="/var/lib/kubelet/pods/5952e0fe-3c89-4ec9-b8b0-79842c388b67/volumes" Nov 02 01:57:24.177912 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:57:24.177873 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:57:28.271942 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:28.271898 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:28.420520 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:28.420474 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:28.698286 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:28.698193 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:33.048300 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:33.048250 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:33.207696 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:33.207643 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:33.355590 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:33.355509 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:38.090847 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:38.090808 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:38.645958 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:38.645925 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:38.869903 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:38.869870 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:43.019707 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:43.019663 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:44.388739 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:44.388696 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:47.775652 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:47.775622 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:50.107841 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:50.107808 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:52.612594 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:52.612563 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:54.175112 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:57:54.175065 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:57:56.154975 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:56.154937 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:56.614631 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:56.614601 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:57.589164 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:57.589132 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:58.851769 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:58.851738 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:57:59.288593 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:57:59.288536 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:00.154873 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:00.154843 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:00.457193 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:00.457110 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:03.184912 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:03.184873 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:04.374757 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:04.374722 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:04.584654 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:04.584587 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:04.768593 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:04.768561 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:08.821023 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:08.820994 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:14.031223 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:14.031183 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:14.208368 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:14.208328 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:14.353757 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:14.353637 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:15.784098 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:15.784049 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:16.892174 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:16.892128 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:17.244111 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:17.243980 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:19.719923 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:19.719890 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:20.253635 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:20.253591 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:20.430100 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:20.430064 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:21.559584 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:21.559545 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:21.770883 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:21.770843 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:22.117351 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:22.117292 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:24.179378 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:58:24.179283 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:58:25.136038 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:25.136001 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:25.291283 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:25.291248 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:25.651915 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:25.651883 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:30.190353 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.190318 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf"] Nov 02 01:58:30.191169 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.191147 5044 cpu_manager.go:414] "containerMap: removing container" podUID="c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" containerName="istio-proxy" Nov 02 01:58:30.191273 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.191171 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" containerName="istio-proxy" Nov 02 01:58:30.191273 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.191233 5044 cpu_manager.go:414] "containerMap: removing container" podUID="e8266d54-e810-4f6d-8720-f903ee12ce08" containerName="istio-proxy" Nov 02 01:58:30.191273 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.191242 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8266d54-e810-4f6d-8720-f903ee12ce08" containerName="istio-proxy" Nov 02 01:58:30.191432 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.191284 5044 cpu_manager.go:414] "containerMap: removing container" podUID="5952e0fe-3c89-4ec9-b8b0-79842c388b67" containerName="istio-proxy" Nov 02 01:58:30.191432 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.191293 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5952e0fe-3c89-4ec9-b8b0-79842c388b67" containerName="istio-proxy" Nov 02 01:58:30.191538 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.191467 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8266d54-e810-4f6d-8720-f903ee12ce08" containerName="istio-proxy" Nov 02 01:58:30.191538 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.191489 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="5952e0fe-3c89-4ec9-b8b0-79842c388b67" containerName="istio-proxy" Nov 02 01:58:30.191538 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.191501 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="c93d45d5-23dc-49ca-a2f4-c41ebdd0c15e" containerName="istio-proxy" Nov 02 01:58:30.195399 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.195376 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" Nov 02 01:58:30.264551 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.264517 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0072c75-d48f-4a49-a1ba-5586e673fd36-metrics-certs\") pod \"maas-api-e2e-api-a-109b07-588c754c4b-mrdxf\" (UID: \"e0072c75-d48f-4a49-a1ba-5586e673fd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" Nov 02 01:58:30.264768 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.264582 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0072c75-d48f-4a49-a1ba-5586e673fd36-openshift-service-ca\") pod \"maas-api-e2e-api-a-109b07-588c754c4b-mrdxf\" (UID: \"e0072c75-d48f-4a49-a1ba-5586e673fd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" Nov 02 01:58:30.264859 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.264836 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5ffv\" (UniqueName: \"kubernetes.io/projected/e0072c75-d48f-4a49-a1ba-5586e673fd36-kube-api-access-w5ffv\") pod \"maas-api-e2e-api-a-109b07-588c754c4b-mrdxf\" (UID: \"e0072c75-d48f-4a49-a1ba-5586e673fd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" Nov 02 01:58:30.264971 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.264949 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0072c75-d48f-4a49-a1ba-5586e673fd36-maas-api-tls\") pod \"maas-api-e2e-api-a-109b07-588c754c4b-mrdxf\" (UID: \"e0072c75-d48f-4a49-a1ba-5586e673fd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" Nov 02 01:58:30.303217 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.303187 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-109b07-serving-cert\"" Nov 02 01:58:30.306812 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.306782 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf"] Nov 02 01:58:30.366122 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.366091 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0072c75-d48f-4a49-a1ba-5586e673fd36-metrics-certs\") pod \"maas-api-e2e-api-a-109b07-588c754c4b-mrdxf\" (UID: \"e0072c75-d48f-4a49-a1ba-5586e673fd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" Nov 02 01:58:30.366288 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.366142 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0072c75-d48f-4a49-a1ba-5586e673fd36-openshift-service-ca\") pod \"maas-api-e2e-api-a-109b07-588c754c4b-mrdxf\" (UID: \"e0072c75-d48f-4a49-a1ba-5586e673fd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" Nov 02 01:58:30.366288 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.366214 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5ffv\" (UniqueName: \"kubernetes.io/projected/e0072c75-d48f-4a49-a1ba-5586e673fd36-kube-api-access-w5ffv\") pod \"maas-api-e2e-api-a-109b07-588c754c4b-mrdxf\" (UID: \"e0072c75-d48f-4a49-a1ba-5586e673fd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" Nov 02 01:58:30.366288 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.366266 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0072c75-d48f-4a49-a1ba-5586e673fd36-maas-api-tls\") pod \"maas-api-e2e-api-a-109b07-588c754c4b-mrdxf\" (UID: \"e0072c75-d48f-4a49-a1ba-5586e673fd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" Nov 02 01:58:30.366478 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:58:30.366460 5044 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-serving-cert: secret "maas-api-e2e-api-a-109b07-serving-cert" not found Nov 02 01:58:30.366551 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:58:30.366528 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0072c75-d48f-4a49-a1ba-5586e673fd36-maas-api-tls podName:e0072c75-d48f-4a49-a1ba-5586e673fd36 nodeName:}" failed. No retries permitted until 2026-11-02 01:58:30.86651097 +0000 UTC m=+2027.537763649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0072c75-d48f-4a49-a1ba-5586e673fd36-maas-api-tls") pod "maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" (UID: "e0072c75-d48f-4a49-a1ba-5586e673fd36") : secret "maas-api-e2e-api-a-109b07-serving-cert" not found Nov 02 01:58:30.366966 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.366943 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0072c75-d48f-4a49-a1ba-5586e673fd36-openshift-service-ca\") pod \"maas-api-e2e-api-a-109b07-588c754c4b-mrdxf\" (UID: \"e0072c75-d48f-4a49-a1ba-5586e673fd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" Nov 02 01:58:30.369836 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.369812 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0072c75-d48f-4a49-a1ba-5586e673fd36-metrics-certs\") pod \"maas-api-e2e-api-a-109b07-588c754c4b-mrdxf\" (UID: \"e0072c75-d48f-4a49-a1ba-5586e673fd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" Nov 02 01:58:30.430073 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.430041 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5ffv\" (UniqueName: \"kubernetes.io/projected/e0072c75-d48f-4a49-a1ba-5586e673fd36-kube-api-access-w5ffv\") pod \"maas-api-e2e-api-a-109b07-588c754c4b-mrdxf\" (UID: \"e0072c75-d48f-4a49-a1ba-5586e673fd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" Nov 02 01:58:30.473036 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.472976 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:30.652628 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.652589 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:30.841752 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.841717 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:30.875368 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.875337 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0072c75-d48f-4a49-a1ba-5586e673fd36-maas-api-tls\") pod \"maas-api-e2e-api-a-109b07-588c754c4b-mrdxf\" (UID: \"e0072c75-d48f-4a49-a1ba-5586e673fd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" Nov 02 01:58:30.877791 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:30.877769 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0072c75-d48f-4a49-a1ba-5586e673fd36-maas-api-tls\") pod \"maas-api-e2e-api-a-109b07-588c754c4b-mrdxf\" (UID: \"e0072c75-d48f-4a49-a1ba-5586e673fd36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" Nov 02 01:58:31.109741 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:31.109649 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" Nov 02 01:58:31.497312 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:31.497286 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf"] Nov 02 01:58:31.500870 ip-10-0-129-6 kubenswrapper[5044]: W1102 01:58:31.500840 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0072c75_d48f_4a49_a1ba_5586e673fd36.slice/crio-edd59f30c65eb685e34e52d6476fea5306a7743a391c6d94e771929feff0fe7e WatchSource:0}: Error finding container edd59f30c65eb685e34e52d6476fea5306a7743a391c6d94e771929feff0fe7e: Status 404 returned error can't find the container with id edd59f30c65eb685e34e52d6476fea5306a7743a391c6d94e771929feff0fe7e Nov 02 01:58:32.211075 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:32.211035 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" event={"ID":"e0072c75-d48f-4a49-a1ba-5586e673fd36","Type":"ContainerStarted","Data":"edd59f30c65eb685e34e52d6476fea5306a7743a391c6d94e771929feff0fe7e"} Nov 02 01:58:33.219992 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:33.219954 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" event={"ID":"e0072c75-d48f-4a49-a1ba-5586e673fd36","Type":"ContainerStarted","Data":"cad05d7289ad4df0895604e7298868435eb1378ef610bcd473703e8b6086edf4"} Nov 02 01:58:33.220397 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:33.220033 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" Nov 02 01:58:33.353970 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:33.353918 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" podStartSLOduration=2.536473871 podStartE2EDuration="3.353904226s" totalImagesPullingTime="817.430355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 01:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 01:58:33.350839709 +0000 UTC m=+2030.022092391" watchObservedRunningTime="2026-11-02 01:58:33.353904226 +0000 UTC m=+2030.025156949" Nov 02 01:58:34.304708 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:34.304640 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:34.493882 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:34.493848 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:35.936987 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:35.936952 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:36.695068 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:36.695034 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:36.900775 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:36.900740 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:39.232742 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:39.232698 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" Nov 02 01:58:40.891789 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:40.891758 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:44.736999 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:44.736954 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:45.665217 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:45.665184 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:50.613577 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:50.613520 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:58:54.176223 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:58:54.176181 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:58:56.184186 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:58:56.184139 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:00.541589 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:00.541553 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:00.692487 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:00.692451 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:00.878425 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:00.878341 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr"] Nov 02 01:59:00.878621 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:00.878598 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" podUID="33422040-a460-438d-a99b-65df8511eeb0" containerName="istio-proxy" containerID="cri-o://5e577852e5c654d4d60d03cf1ff74476c5bfaebb3f59ec37eceede6e403e239d" gracePeriod=30 Nov 02 01:59:01.295754 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:01.295720 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:04.562330 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:04.562286 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 01:59:04.562782 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:04.562351 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" podUID="33422040-a460-438d-a99b-65df8511eeb0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 01:59:06.332778 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.328620 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:06.559377 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.559342 5044 generic.go:356] "Generic (PLEG): container finished" podID="33422040-a460-438d-a99b-65df8511eeb0" containerID="5e577852e5c654d4d60d03cf1ff74476c5bfaebb3f59ec37eceede6e403e239d" exitCode=0 Nov 02 01:59:06.559614 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.559597 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" event={"ID":"33422040-a460-438d-a99b-65df8511eeb0","Type":"ContainerDied","Data":"5e577852e5c654d4d60d03cf1ff74476c5bfaebb3f59ec37eceede6e403e239d"} Nov 02 01:59:06.630939 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.630911 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:59:06.691805 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.691772 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33422040-a460-438d-a99b-65df8511eeb0-kube-api-access-grgwl\" (UniqueName: \"kubernetes.io/projected/33422040-a460-438d-a99b-65df8511eeb0-kube-api-access-grgwl\") pod \"33422040-a460-438d-a99b-65df8511eeb0\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " Nov 02 01:59:06.691973 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.691837 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-credential-socket\") pod \"33422040-a460-438d-a99b-65df8511eeb0\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " Nov 02 01:59:06.691973 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.691906 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33422040-a460-438d-a99b-65df8511eeb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33422040-a460-438d-a99b-65df8511eeb0-istio-podinfo\") pod \"33422040-a460-438d-a99b-65df8511eeb0\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " Nov 02 01:59:06.691973 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.691966 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-istio-data\") pod \"33422040-a460-438d-a99b-65df8511eeb0\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " Nov 02 01:59:06.692164 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.692069 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33422040-a460-438d-a99b-65df8511eeb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33422040-a460-438d-a99b-65df8511eeb0-istiod-ca-cert\") pod \"33422040-a460-438d-a99b-65df8511eeb0\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " Nov 02 01:59:06.692164 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.692143 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-istio-envoy\") pod \"33422040-a460-438d-a99b-65df8511eeb0\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " Nov 02 01:59:06.692274 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.692171 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-credential-socket" pod "33422040-a460-438d-a99b-65df8511eeb0" (UID: "33422040-a460-438d-a99b-65df8511eeb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:59:06.692274 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.692245 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-workload-certs\") pod \"33422040-a460-438d-a99b-65df8511eeb0\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " Nov 02 01:59:06.692383 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.692282 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-workload-socket\") pod \"33422040-a460-438d-a99b-65df8511eeb0\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " Nov 02 01:59:06.692383 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.692337 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33422040-a460-438d-a99b-65df8511eeb0-istio-token\" (UniqueName: \"kubernetes.io/projected/33422040-a460-438d-a99b-65df8511eeb0-istio-token\") pod \"33422040-a460-438d-a99b-65df8511eeb0\" (UID: \"33422040-a460-438d-a99b-65df8511eeb0\") " Nov 02 01:59:06.692487 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.692449 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-istio-data" pod "33422040-a460-438d-a99b-65df8511eeb0" (UID: "33422040-a460-438d-a99b-65df8511eeb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:59:06.692644 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.692614 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-workload-socket" pod "33422040-a460-438d-a99b-65df8511eeb0" (UID: "33422040-a460-438d-a99b-65df8511eeb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:59:06.693045 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.693021 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-workload-certs" pod "33422040-a460-438d-a99b-65df8511eeb0" (UID: "33422040-a460-438d-a99b-65df8511eeb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:59:06.693136 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.693051 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33422040-a460-438d-a99b-65df8511eeb0-istiod-ca-cert" pod "33422040-a460-438d-a99b-65df8511eeb0" (UID: "33422040-a460-438d-a99b-65df8511eeb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 01:59:06.693197 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.693169 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:59:06.693197 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.693191 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:59:06.693306 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.693207 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:59:06.694586 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.694556 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33422040-a460-438d-a99b-65df8511eeb0-kube-api-access-grgwl" pod "33422040-a460-438d-a99b-65df8511eeb0" (UID: "33422040-a460-438d-a99b-65df8511eeb0"). InnerVolumeSpecName "kube-api-access-grgwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:59:06.694718 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.694621 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-istio-envoy" pod "33422040-a460-438d-a99b-65df8511eeb0" (UID: "33422040-a460-438d-a99b-65df8511eeb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 01:59:06.695164 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.695133 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33422040-a460-438d-a99b-65df8511eeb0-istio-token" pod "33422040-a460-438d-a99b-65df8511eeb0" (UID: "33422040-a460-438d-a99b-65df8511eeb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 01:59:06.695330 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.695311 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33422040-a460-438d-a99b-65df8511eeb0-istio-podinfo" pod "33422040-a460-438d-a99b-65df8511eeb0" (UID: "33422040-a460-438d-a99b-65df8511eeb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 01:59:06.794514 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.794471 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:59:06.794514 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.794505 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33422040-a460-438d-a99b-65df8511eeb0-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:59:06.794514 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.794519 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grgwl\" (UniqueName: \"kubernetes.io/projected/33422040-a460-438d-a99b-65df8511eeb0-kube-api-access-grgwl\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:59:06.794765 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.794534 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33422040-a460-438d-a99b-65df8511eeb0-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:59:06.794765 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.794544 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33422040-a460-438d-a99b-65df8511eeb0-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:59:06.794765 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:06.794552 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33422040-a460-438d-a99b-65df8511eeb0-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 01:59:07.568288 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:07.567624 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" Nov 02 01:59:07.568288 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:07.567652 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr" event={"ID":"33422040-a460-438d-a99b-65df8511eeb0","Type":"ContainerDied","Data":"6af1911e87de79cba884136297b8bbf8af62136b81431af96c978a85d5471fdd"} Nov 02 01:59:07.568288 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:07.567752 5044 scope.go:122] "RemoveContainer" containerID="5e577852e5c654d4d60d03cf1ff74476c5bfaebb3f59ec37eceede6e403e239d" Nov 02 01:59:07.791763 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:07.791727 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr"] Nov 02 01:59:07.840456 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:07.840426 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-mqchr"] Nov 02 01:59:09.454153 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:09.454120 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33422040-a460-438d-a99b-65df8511eeb0" path="/var/lib/kubelet/pods/33422040-a460-438d-a99b-65df8511eeb0/volumes" Nov 02 01:59:11.051064 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:11.051027 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:11.950279 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:11.950254 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:12.195914 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:12.195883 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:12.957323 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:12.957272 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:13.323610 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:13.323578 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:14.486833 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:14.486806 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:14.658850 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:14.658806 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:17.269860 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:17.269824 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:17.416267 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:17.416234 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:17.569439 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:17.569308 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:20.569112 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:20.569061 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:21.227830 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:21.226937 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:21.578025 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:21.577989 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:22.676164 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:22.676133 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:24.174617 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:59:24.174586 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:59:25.361485 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:25.361448 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:25.541664 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:25.541630 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:25.804820 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:25.804768 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:27.717225 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:27.717194 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:32.685822 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:32.685772 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:32.844488 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:32.844454 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:32.998309 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:32.996898 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:38.027842 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:38.027805 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:38.562881 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:38.562843 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:38.755715 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:38.755621 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:42.977024 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:42.976987 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:43.726401 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:43.726371 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 01:59:43.730082 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:43.730056 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 01:59:43.849907 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:43.849879 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:43.993834 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:43.993751 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:44.660014 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:44.659974 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:46.447723 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:46.447694 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:47.774711 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:47.774661 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:52.706477 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:52.706398 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:52.852563 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:52.852528 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:53.000422 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:53.000327 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:54.177933 ip-10-0-129-6 kubenswrapper[5044]: E1102 01:59:54.177895 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 01:59:58.051481 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:58.051450 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:59.038875 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:59.038840 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 01:59:59.257366 ip-10-0-129-6 kubenswrapper[5044]: I1102 01:59:59.257316 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:00.201295 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.201255 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb"] Nov 02 02:00:00.202325 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.202303 5044 cpu_manager.go:414] "containerMap: removing container" podUID="33422040-a460-438d-a99b-65df8511eeb0" containerName="istio-proxy" Nov 02 02:00:00.202400 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.202327 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33422040-a460-438d-a99b-65df8511eeb0" containerName="istio-proxy" Nov 02 02:00:00.202534 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.202522 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="33422040-a460-438d-a99b-65df8511eeb0" containerName="istio-proxy" Nov 02 02:00:00.209159 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.209135 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" Nov 02 02:00:00.224372 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.224347 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q"] Nov 02 02:00:00.228260 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.228240 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" Nov 02 02:00:00.239376 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.239354 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l"] Nov 02 02:00:00.243171 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.243155 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" Nov 02 02:00:00.251408 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.251377 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f"] Nov 02 02:00:00.255313 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.255296 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" Nov 02 02:00:00.307119 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.307082 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kgg8\" (UniqueName: \"kubernetes.io/projected/c347ed5b-42dc-460d-97a4-bb0f386c30ed-kube-api-access-9kgg8\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f\" (UID: \"c347ed5b-42dc-460d-97a4-bb0f386c30ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" Nov 02 02:00:00.307507 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.307131 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c3b60b-7def-49be-a219-34f221155180-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l\" (UID: \"a2c3b60b-7def-49be-a219-34f221155180\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" Nov 02 02:00:00.307507 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.307233 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqp7h\" (UniqueName: \"kubernetes.io/projected/a2c3b60b-7def-49be-a219-34f221155180-kube-api-access-wqp7h\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l\" (UID: \"a2c3b60b-7def-49be-a219-34f221155180\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" Nov 02 02:00:00.307507 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.307359 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c2d5cf9-e281-475c-a401-887f631f9094-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q\" (UID: \"4c2d5cf9-e281-475c-a401-887f631f9094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" Nov 02 02:00:00.307699 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.307554 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a10a923-11d9-4a35-a277-1ca557bdbf53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb\" (UID: \"9a10a923-11d9-4a35-a277-1ca557bdbf53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" Nov 02 02:00:00.307699 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.307627 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c347ed5b-42dc-460d-97a4-bb0f386c30ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f\" (UID: \"c347ed5b-42dc-460d-97a4-bb0f386c30ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" Nov 02 02:00:00.307828 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.307740 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nh8r\" (UniqueName: \"kubernetes.io/projected/4c2d5cf9-e281-475c-a401-887f631f9094-kube-api-access-2nh8r\") pod \"maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q\" (UID: \"4c2d5cf9-e281-475c-a401-887f631f9094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" Nov 02 02:00:00.308498 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.308039 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-789nj\" (UniqueName: \"kubernetes.io/projected/9a10a923-11d9-4a35-a277-1ca557bdbf53-kube-api-access-789nj\") pod \"maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb\" (UID: \"9a10a923-11d9-4a35-a277-1ca557bdbf53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" Nov 02 02:00:00.313961 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.313936 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb"] Nov 02 02:00:00.325288 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.325251 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l"] Nov 02 02:00:00.336091 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.336016 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f"] Nov 02 02:00:00.362477 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.362451 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\"" Nov 02 02:00:00.385731 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.385697 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q"] Nov 02 02:00:00.409588 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.409553 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c2d5cf9-e281-475c-a401-887f631f9094-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q\" (UID: \"4c2d5cf9-e281-475c-a401-887f631f9094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" Nov 02 02:00:00.409802 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.409613 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a10a923-11d9-4a35-a277-1ca557bdbf53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb\" (UID: \"9a10a923-11d9-4a35-a277-1ca557bdbf53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" Nov 02 02:00:00.409802 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.409735 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c347ed5b-42dc-460d-97a4-bb0f386c30ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f\" (UID: \"c347ed5b-42dc-460d-97a4-bb0f386c30ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" Nov 02 02:00:00.409802 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.409774 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nh8r\" (UniqueName: \"kubernetes.io/projected/4c2d5cf9-e281-475c-a401-887f631f9094-kube-api-access-2nh8r\") pod \"maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q\" (UID: \"4c2d5cf9-e281-475c-a401-887f631f9094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" Nov 02 02:00:00.409999 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.409891 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-789nj\" (UniqueName: \"kubernetes.io/projected/9a10a923-11d9-4a35-a277-1ca557bdbf53-kube-api-access-789nj\") pod \"maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb\" (UID: \"9a10a923-11d9-4a35-a277-1ca557bdbf53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" Nov 02 02:00:00.410057 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.410016 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kgg8\" (UniqueName: \"kubernetes.io/projected/c347ed5b-42dc-460d-97a4-bb0f386c30ed-kube-api-access-9kgg8\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f\" (UID: \"c347ed5b-42dc-460d-97a4-bb0f386c30ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" Nov 02 02:00:00.410057 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.410046 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c3b60b-7def-49be-a219-34f221155180-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l\" (UID: \"a2c3b60b-7def-49be-a219-34f221155180\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" Nov 02 02:00:00.410172 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.410079 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqp7h\" (UniqueName: \"kubernetes.io/projected/a2c3b60b-7def-49be-a219-34f221155180-kube-api-access-wqp7h\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l\" (UID: \"a2c3b60b-7def-49be-a219-34f221155180\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" Nov 02 02:00:00.410493 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.410446 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a10a923-11d9-4a35-a277-1ca557bdbf53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb\" (UID: \"9a10a923-11d9-4a35-a277-1ca557bdbf53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" Nov 02 02:00:00.410608 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.410514 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c2d5cf9-e281-475c-a401-887f631f9094-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q\" (UID: \"4c2d5cf9-e281-475c-a401-887f631f9094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" Nov 02 02:00:00.410608 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.410528 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c347ed5b-42dc-460d-97a4-bb0f386c30ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f\" (UID: \"c347ed5b-42dc-460d-97a4-bb0f386c30ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" Nov 02 02:00:00.410894 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.410874 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c3b60b-7def-49be-a219-34f221155180-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l\" (UID: \"a2c3b60b-7def-49be-a219-34f221155180\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" Nov 02 02:00:00.472144 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.472067 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nh8r\" (UniqueName: \"kubernetes.io/projected/4c2d5cf9-e281-475c-a401-887f631f9094-kube-api-access-2nh8r\") pod \"maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q\" (UID: \"4c2d5cf9-e281-475c-a401-887f631f9094\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" Nov 02 02:00:00.472924 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.472878 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kgg8\" (UniqueName: \"kubernetes.io/projected/c347ed5b-42dc-460d-97a4-bb0f386c30ed-kube-api-access-9kgg8\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f\" (UID: \"c347ed5b-42dc-460d-97a4-bb0f386c30ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" Nov 02 02:00:00.473053 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.472880 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-789nj\" (UniqueName: \"kubernetes.io/projected/9a10a923-11d9-4a35-a277-1ca557bdbf53-kube-api-access-789nj\") pod \"maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb\" (UID: \"9a10a923-11d9-4a35-a277-1ca557bdbf53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" Nov 02 02:00:00.473809 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.473785 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqp7h\" (UniqueName: \"kubernetes.io/projected/a2c3b60b-7def-49be-a219-34f221155180-kube-api-access-wqp7h\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l\" (UID: \"a2c3b60b-7def-49be-a219-34f221155180\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" Nov 02 02:00:00.524594 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.524564 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" Nov 02 02:00:00.542046 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.542017 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" Nov 02 02:00:00.558515 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.558487 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" Nov 02 02:00:00.574716 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.571871 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" Nov 02 02:00:00.916540 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.916509 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb"] Nov 02 02:00:00.917045 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:00:00.917017 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a10a923_11d9_4a35_a277_1ca557bdbf53.slice/crio-c4f9b23e83f9dca0053ad35403932b03983eaff58fee9741d0c288fb1cc25058 WatchSource:0}: Error finding container c4f9b23e83f9dca0053ad35403932b03983eaff58fee9741d0c288fb1cc25058: Status 404 returned error can't find the container with id c4f9b23e83f9dca0053ad35403932b03983eaff58fee9741d0c288fb1cc25058 Nov 02 02:00:00.968793 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:00.968766 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q"] Nov 02 02:00:00.969318 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:00:00.969291 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2d5cf9_e281_475c_a401_887f631f9094.slice/crio-56173034a55e89fe55b216b2e94588862d94d944d0f2ed50b2c6f2898457105f WatchSource:0}: Error finding container 56173034a55e89fe55b216b2e94588862d94d944d0f2ed50b2c6f2898457105f: Status 404 returned error can't find the container with id 56173034a55e89fe55b216b2e94588862d94d944d0f2ed50b2c6f2898457105f Nov 02 02:00:01.022516 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:00:01.022481 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c3b60b_7def_49be_a219_34f221155180.slice/crio-95379fdac2d6b38784bc3ab630c85821e92d38fd48d217453bac283f9da943f6 WatchSource:0}: Error finding container 95379fdac2d6b38784bc3ab630c85821e92d38fd48d217453bac283f9da943f6: Status 404 returned error can't find the container with id 95379fdac2d6b38784bc3ab630c85821e92d38fd48d217453bac283f9da943f6 Nov 02 02:00:01.022809 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:01.022706 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l"] Nov 02 02:00:01.023261 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:00:01.023241 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc347ed5b_42dc_460d_97a4_bb0f386c30ed.slice/crio-43b63d65468bfd3b78e4569f462daa41d8eeaffb1ac2e7531a67b39491705d75 WatchSource:0}: Error finding container 43b63d65468bfd3b78e4569f462daa41d8eeaffb1ac2e7531a67b39491705d75: Status 404 returned error can't find the container with id 43b63d65468bfd3b78e4569f462daa41d8eeaffb1ac2e7531a67b39491705d75 Nov 02 02:00:01.026523 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:01.026495 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f"] Nov 02 02:00:01.042375 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:01.042350 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" event={"ID":"9a10a923-11d9-4a35-a277-1ca557bdbf53","Type":"ContainerStarted","Data":"c4f9b23e83f9dca0053ad35403932b03983eaff58fee9741d0c288fb1cc25058"} Nov 02 02:00:01.043857 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:01.043827 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" event={"ID":"c347ed5b-42dc-460d-97a4-bb0f386c30ed","Type":"ContainerStarted","Data":"43b63d65468bfd3b78e4569f462daa41d8eeaffb1ac2e7531a67b39491705d75"} Nov 02 02:00:01.045456 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:01.045427 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" event={"ID":"a2c3b60b-7def-49be-a219-34f221155180","Type":"ContainerStarted","Data":"95379fdac2d6b38784bc3ab630c85821e92d38fd48d217453bac283f9da943f6"} Nov 02 02:00:01.046967 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:01.046946 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" event={"ID":"4c2d5cf9-e281-475c-a401-887f631f9094","Type":"ContainerStarted","Data":"56173034a55e89fe55b216b2e94588862d94d944d0f2ed50b2c6f2898457105f"} Nov 02 02:00:02.055919 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:02.055815 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" event={"ID":"a2c3b60b-7def-49be-a219-34f221155180","Type":"ContainerStarted","Data":"aec2f4852f6a19520568213d9ddd5a7a172a3bbcd8ef18b0d65015870b40242d"} Nov 02 02:00:02.057591 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:02.057558 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" event={"ID":"4c2d5cf9-e281-475c-a401-887f631f9094","Type":"ContainerStarted","Data":"c85b202aa3a99808f37e268eaf4b19d51b90825af8287867d6f03082e1e97475"} Nov 02 02:00:02.059252 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:02.059231 5044 generic.go:356] "Generic (PLEG): container finished" podID="9a10a923-11d9-4a35-a277-1ca557bdbf53" containerID="9fd54487c8317dc2fa5641e58df1cb4361ee48ad37bd5986869cd511e3fef79e" exitCode=60 Nov 02 02:00:02.059362 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:02.059326 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" event={"ID":"9a10a923-11d9-4a35-a277-1ca557bdbf53","Type":"ContainerDied","Data":"9fd54487c8317dc2fa5641e58df1cb4361ee48ad37bd5986869cd511e3fef79e"} Nov 02 02:00:02.059536 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:02.059520 5044 scope.go:122] "RemoveContainer" containerID="9fd54487c8317dc2fa5641e58df1cb4361ee48ad37bd5986869cd511e3fef79e" Nov 02 02:00:02.061012 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:02.060986 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" event={"ID":"c347ed5b-42dc-460d-97a4-bb0f386c30ed","Type":"ContainerStarted","Data":"7e1ab45a6edf6c27e51c31ac39180ec69b70898f95642a48cad075e05510f4ee"} Nov 02 02:00:02.184772 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:02.184708 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" podStartSLOduration=2.184688044 podStartE2EDuration="2.184688044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:02.178910047 +0000 UTC m=+2118.850162729" watchObservedRunningTime="2026-11-02 02:00:02.184688044 +0000 UTC m=+2118.855940782" Nov 02 02:00:02.416451 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:02.416349 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" podStartSLOduration=2.416335356 podStartE2EDuration="2.416335356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:02.412986862 +0000 UTC m=+2119.084239544" watchObservedRunningTime="2026-11-02 02:00:02.416335356 +0000 UTC m=+2119.087588061" Nov 02 02:00:02.530423 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:02.530375 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" podStartSLOduration=2.530359986 podStartE2EDuration="2.530359986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:02.529514469 +0000 UTC m=+2119.200767151" watchObservedRunningTime="2026-11-02 02:00:02.530359986 +0000 UTC m=+2119.201612666" Nov 02 02:00:03.072735 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:03.072689 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" event={"ID":"9a10a923-11d9-4a35-a277-1ca557bdbf53","Type":"ContainerStarted","Data":"a1209e708f4ab6720f17439e7be0bc5af4b5ae6bdfadae14c93942e0c6ed17ca"} Nov 02 02:00:03.074483 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:03.074456 5044 generic.go:356] "Generic (PLEG): container finished" podID="c347ed5b-42dc-460d-97a4-bb0f386c30ed" containerID="7e1ab45a6edf6c27e51c31ac39180ec69b70898f95642a48cad075e05510f4ee" exitCode=60 Nov 02 02:00:03.074610 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:03.074534 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" event={"ID":"c347ed5b-42dc-460d-97a4-bb0f386c30ed","Type":"ContainerDied","Data":"7e1ab45a6edf6c27e51c31ac39180ec69b70898f95642a48cad075e05510f4ee"} Nov 02 02:00:03.074818 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:03.074802 5044 scope.go:122] "RemoveContainer" containerID="7e1ab45a6edf6c27e51c31ac39180ec69b70898f95642a48cad075e05510f4ee" Nov 02 02:00:03.076424 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:03.076393 5044 generic.go:356] "Generic (PLEG): container finished" podID="a2c3b60b-7def-49be-a219-34f221155180" containerID="aec2f4852f6a19520568213d9ddd5a7a172a3bbcd8ef18b0d65015870b40242d" exitCode=60 Nov 02 02:00:03.076526 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:03.076474 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" event={"ID":"a2c3b60b-7def-49be-a219-34f221155180","Type":"ContainerDied","Data":"aec2f4852f6a19520568213d9ddd5a7a172a3bbcd8ef18b0d65015870b40242d"} Nov 02 02:00:03.076726 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:03.076711 5044 scope.go:122] "RemoveContainer" containerID="aec2f4852f6a19520568213d9ddd5a7a172a3bbcd8ef18b0d65015870b40242d" Nov 02 02:00:03.078978 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:03.078961 5044 generic.go:356] "Generic (PLEG): container finished" podID="4c2d5cf9-e281-475c-a401-887f631f9094" containerID="c85b202aa3a99808f37e268eaf4b19d51b90825af8287867d6f03082e1e97475" exitCode=60 Nov 02 02:00:03.079085 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:03.079071 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" event={"ID":"4c2d5cf9-e281-475c-a401-887f631f9094","Type":"ContainerDied","Data":"c85b202aa3a99808f37e268eaf4b19d51b90825af8287867d6f03082e1e97475"} Nov 02 02:00:03.079275 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:03.079257 5044 scope.go:122] "RemoveContainer" containerID="c85b202aa3a99808f37e268eaf4b19d51b90825af8287867d6f03082e1e97475" Nov 02 02:00:03.194604 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:03.194547 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" podStartSLOduration=3.194533526 podStartE2EDuration="3.194533526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:03.192761963 +0000 UTC m=+2119.864014644" watchObservedRunningTime="2026-11-02 02:00:03.194533526 +0000 UTC m=+2119.865786206" Nov 02 02:00:03.236186 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:03.236149 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:04.089154 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:04.089123 5044 generic.go:356] "Generic (PLEG): container finished" podID="9a10a923-11d9-4a35-a277-1ca557bdbf53" containerID="a1209e708f4ab6720f17439e7be0bc5af4b5ae6bdfadae14c93942e0c6ed17ca" exitCode=60 Nov 02 02:00:04.089566 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:04.089212 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" event={"ID":"9a10a923-11d9-4a35-a277-1ca557bdbf53","Type":"ContainerDied","Data":"a1209e708f4ab6720f17439e7be0bc5af4b5ae6bdfadae14c93942e0c6ed17ca"} Nov 02 02:00:04.089566 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:04.089281 5044 scope.go:122] "RemoveContainer" containerID="9fd54487c8317dc2fa5641e58df1cb4361ee48ad37bd5986869cd511e3fef79e" Nov 02 02:00:04.089694 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:04.089660 5044 scope.go:122] "RemoveContainer" containerID="a1209e708f4ab6720f17439e7be0bc5af4b5ae6bdfadae14c93942e0c6ed17ca" Nov 02 02:00:04.090106 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:00:04.090080 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb_odh-ai-gateway-infra(9a10a923-11d9-4a35-a277-1ca557bdbf53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" podUID="9a10a923-11d9-4a35-a277-1ca557bdbf53" Nov 02 02:00:04.091496 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:04.091474 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" event={"ID":"c347ed5b-42dc-460d-97a4-bb0f386c30ed","Type":"ContainerStarted","Data":"278f8d58fbbca1d8e24cc5ea05992679ce64c153d838b3ae66f07ce2786454fd"} Nov 02 02:00:04.093774 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:04.093750 5044 generic.go:356] "Generic (PLEG): container finished" podID="a2c3b60b-7def-49be-a219-34f221155180" containerID="2e391f43da12606dc7388793f1acce945a36316b2960cd573626b8475297963e" exitCode=60 Nov 02 02:00:04.093872 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:04.093836 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" event={"ID":"a2c3b60b-7def-49be-a219-34f221155180","Type":"ContainerDied","Data":"2e391f43da12606dc7388793f1acce945a36316b2960cd573626b8475297963e"} Nov 02 02:00:04.094206 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:04.094136 5044 scope.go:122] "RemoveContainer" containerID="2e391f43da12606dc7388793f1acce945a36316b2960cd573626b8475297963e" Nov 02 02:00:04.094505 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:00:04.094481 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l_odh-ai-gateway-infra(a2c3b60b-7def-49be-a219-34f221155180)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" podUID="a2c3b60b-7def-49be-a219-34f221155180" Nov 02 02:00:04.096232 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:04.096211 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" event={"ID":"4c2d5cf9-e281-475c-a401-887f631f9094","Type":"ContainerStarted","Data":"81c22f79d2cff07510a1d1938b94ef78cf9bb136aa0222004b768f96601c01f9"} Nov 02 02:00:04.131698 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:04.131665 5044 scope.go:122] "RemoveContainer" containerID="aec2f4852f6a19520568213d9ddd5a7a172a3bbcd8ef18b0d65015870b40242d" Nov 02 02:00:05.105240 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:05.105210 5044 scope.go:122] "RemoveContainer" containerID="2e391f43da12606dc7388793f1acce945a36316b2960cd573626b8475297963e" Nov 02 02:00:05.105730 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:00:05.105640 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l_odh-ai-gateway-infra(a2c3b60b-7def-49be-a219-34f221155180)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" podUID="a2c3b60b-7def-49be-a219-34f221155180" Nov 02 02:00:05.107300 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:05.107273 5044 generic.go:356] "Generic (PLEG): container finished" podID="4c2d5cf9-e281-475c-a401-887f631f9094" containerID="81c22f79d2cff07510a1d1938b94ef78cf9bb136aa0222004b768f96601c01f9" exitCode=60 Nov 02 02:00:05.107401 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:05.107363 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" event={"ID":"4c2d5cf9-e281-475c-a401-887f631f9094","Type":"ContainerDied","Data":"81c22f79d2cff07510a1d1938b94ef78cf9bb136aa0222004b768f96601c01f9"} Nov 02 02:00:05.107475 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:05.107461 5044 scope.go:122] "RemoveContainer" containerID="c85b202aa3a99808f37e268eaf4b19d51b90825af8287867d6f03082e1e97475" Nov 02 02:00:05.107649 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:05.107633 5044 scope.go:122] "RemoveContainer" containerID="81c22f79d2cff07510a1d1938b94ef78cf9bb136aa0222004b768f96601c01f9" Nov 02 02:00:05.107996 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:00:05.107971 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q_odh-ai-gateway-infra(4c2d5cf9-e281-475c-a401-887f631f9094)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" podUID="4c2d5cf9-e281-475c-a401-887f631f9094" Nov 02 02:00:05.109808 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:05.109791 5044 scope.go:122] "RemoveContainer" containerID="a1209e708f4ab6720f17439e7be0bc5af4b5ae6bdfadae14c93942e0c6ed17ca" Nov 02 02:00:05.110104 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:00:05.110071 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb_odh-ai-gateway-infra(9a10a923-11d9-4a35-a277-1ca557bdbf53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" podUID="9a10a923-11d9-4a35-a277-1ca557bdbf53" Nov 02 02:00:05.111316 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:05.111291 5044 generic.go:356] "Generic (PLEG): container finished" podID="c347ed5b-42dc-460d-97a4-bb0f386c30ed" containerID="278f8d58fbbca1d8e24cc5ea05992679ce64c153d838b3ae66f07ce2786454fd" exitCode=60 Nov 02 02:00:05.111411 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:05.111343 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" event={"ID":"c347ed5b-42dc-460d-97a4-bb0f386c30ed","Type":"ContainerDied","Data":"278f8d58fbbca1d8e24cc5ea05992679ce64c153d838b3ae66f07ce2786454fd"} Nov 02 02:00:05.111545 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:05.111529 5044 scope.go:122] "RemoveContainer" containerID="278f8d58fbbca1d8e24cc5ea05992679ce64c153d838b3ae66f07ce2786454fd" Nov 02 02:00:05.111817 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:00:05.111797 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f_odh-ai-gateway-infra(c347ed5b-42dc-460d-97a4-bb0f386c30ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" podUID="c347ed5b-42dc-460d-97a4-bb0f386c30ed" Nov 02 02:00:05.120802 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:05.120776 5044 scope.go:122] "RemoveContainer" containerID="7e1ab45a6edf6c27e51c31ac39180ec69b70898f95642a48cad075e05510f4ee" Nov 02 02:00:05.906413 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:05.906374 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:06.119655 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:06.119627 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:06.121003 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:06.120985 5044 scope.go:122] "RemoveContainer" containerID="278f8d58fbbca1d8e24cc5ea05992679ce64c153d838b3ae66f07ce2786454fd" Nov 02 02:00:06.121394 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:00:06.121367 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f_odh-ai-gateway-infra(c347ed5b-42dc-460d-97a4-bb0f386c30ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" podUID="c347ed5b-42dc-460d-97a4-bb0f386c30ed" Nov 02 02:00:06.122865 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:06.122823 5044 scope.go:122] "RemoveContainer" containerID="81c22f79d2cff07510a1d1938b94ef78cf9bb136aa0222004b768f96601c01f9" Nov 02 02:00:06.123146 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:00:06.123125 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q_odh-ai-gateway-infra(4c2d5cf9-e281-475c-a401-887f631f9094)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" podUID="4c2d5cf9-e281-475c-a401-887f631f9094" Nov 02 02:00:08.624044 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:08.623990 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:13.898344 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:13.898294 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:15.445292 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:15.445258 5044 scope.go:122] "RemoveContainer" containerID="81c22f79d2cff07510a1d1938b94ef78cf9bb136aa0222004b768f96601c01f9" Nov 02 02:00:16.249936 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:16.249853 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" event={"ID":"4c2d5cf9-e281-475c-a401-887f631f9094","Type":"ContainerStarted","Data":"be0fc791352f51b2aad60d4fcc1745fc5b8e8b106f04671d53cb6c55e9911bef"} Nov 02 02:00:16.444729 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:16.444699 5044 scope.go:122] "RemoveContainer" containerID="278f8d58fbbca1d8e24cc5ea05992679ce64c153d838b3ae66f07ce2786454fd" Nov 02 02:00:16.444903 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:16.444799 5044 scope.go:122] "RemoveContainer" containerID="a1209e708f4ab6720f17439e7be0bc5af4b5ae6bdfadae14c93942e0c6ed17ca" Nov 02 02:00:17.258577 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:17.258531 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" event={"ID":"9a10a923-11d9-4a35-a277-1ca557bdbf53","Type":"ContainerStarted","Data":"269bb8748b9c580bd3a20d9311fd6b28e0c21011f374dab2381d12d72dfecfb7"} Nov 02 02:00:17.260535 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:17.260512 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" event={"ID":"c347ed5b-42dc-460d-97a4-bb0f386c30ed","Type":"ContainerStarted","Data":"61c0a1ab9a0f7d9e1c89afa4e6ec9707bd6abacc94572982e92ab9e6e4c1560a"} Nov 02 02:00:17.263499 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:17.263474 5044 generic.go:356] "Generic (PLEG): container finished" podID="4c2d5cf9-e281-475c-a401-887f631f9094" containerID="be0fc791352f51b2aad60d4fcc1745fc5b8e8b106f04671d53cb6c55e9911bef" exitCode=60 Nov 02 02:00:17.263620 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:17.263545 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" event={"ID":"4c2d5cf9-e281-475c-a401-887f631f9094","Type":"ContainerDied","Data":"be0fc791352f51b2aad60d4fcc1745fc5b8e8b106f04671d53cb6c55e9911bef"} Nov 02 02:00:17.263620 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:17.263597 5044 scope.go:122] "RemoveContainer" containerID="81c22f79d2cff07510a1d1938b94ef78cf9bb136aa0222004b768f96601c01f9" Nov 02 02:00:17.263925 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:17.263908 5044 scope.go:122] "RemoveContainer" containerID="be0fc791352f51b2aad60d4fcc1745fc5b8e8b106f04671d53cb6c55e9911bef" Nov 02 02:00:17.264215 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:00:17.264192 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q_odh-ai-gateway-infra(4c2d5cf9-e281-475c-a401-887f631f9094)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" podUID="4c2d5cf9-e281-475c-a401-887f631f9094" Nov 02 02:00:17.379231 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:17.379201 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q"] Nov 02 02:00:17.445001 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:17.444962 5044 scope.go:122] "RemoveContainer" containerID="2e391f43da12606dc7388793f1acce945a36316b2960cd573626b8475297963e" Nov 02 02:00:18.273653 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:18.273615 5044 generic.go:356] "Generic (PLEG): container finished" podID="9a10a923-11d9-4a35-a277-1ca557bdbf53" containerID="269bb8748b9c580bd3a20d9311fd6b28e0c21011f374dab2381d12d72dfecfb7" exitCode=60 Nov 02 02:00:18.274185 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:18.273705 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" event={"ID":"9a10a923-11d9-4a35-a277-1ca557bdbf53","Type":"ContainerDied","Data":"269bb8748b9c580bd3a20d9311fd6b28e0c21011f374dab2381d12d72dfecfb7"} Nov 02 02:00:18.274185 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:18.273790 5044 scope.go:122] "RemoveContainer" containerID="a1209e708f4ab6720f17439e7be0bc5af4b5ae6bdfadae14c93942e0c6ed17ca" Nov 02 02:00:18.274310 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:18.274295 5044 scope.go:122] "RemoveContainer" containerID="269bb8748b9c580bd3a20d9311fd6b28e0c21011f374dab2381d12d72dfecfb7" Nov 02 02:00:18.274730 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:00:18.274703 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb_odh-ai-gateway-infra(9a10a923-11d9-4a35-a277-1ca557bdbf53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" podUID="9a10a923-11d9-4a35-a277-1ca557bdbf53" Nov 02 02:00:18.275842 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:18.275817 5044 generic.go:356] "Generic (PLEG): container finished" podID="c347ed5b-42dc-460d-97a4-bb0f386c30ed" containerID="61c0a1ab9a0f7d9e1c89afa4e6ec9707bd6abacc94572982e92ab9e6e4c1560a" exitCode=60 Nov 02 02:00:18.275964 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:18.275934 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" event={"ID":"c347ed5b-42dc-460d-97a4-bb0f386c30ed","Type":"ContainerDied","Data":"61c0a1ab9a0f7d9e1c89afa4e6ec9707bd6abacc94572982e92ab9e6e4c1560a"} Nov 02 02:00:18.276281 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:18.276261 5044 scope.go:122] "RemoveContainer" containerID="61c0a1ab9a0f7d9e1c89afa4e6ec9707bd6abacc94572982e92ab9e6e4c1560a" Nov 02 02:00:18.276632 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:00:18.276605 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f_odh-ai-gateway-infra(c347ed5b-42dc-460d-97a4-bb0f386c30ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" podUID="c347ed5b-42dc-460d-97a4-bb0f386c30ed" Nov 02 02:00:18.277994 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:18.277963 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" event={"ID":"a2c3b60b-7def-49be-a219-34f221155180","Type":"ContainerStarted","Data":"6dda7add46aae52530f75a6cdd4f8e60941d8e71f602a48eee894f92e86af1fd"} Nov 02 02:00:18.288284 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:18.288258 5044 scope.go:122] "RemoveContainer" containerID="278f8d58fbbca1d8e24cc5ea05992679ce64c153d838b3ae66f07ce2786454fd" Nov 02 02:00:18.389453 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:18.389420 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb"] Nov 02 02:00:18.471004 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:18.470970 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" Nov 02 02:00:18.506853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:18.506823 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c2d5cf9-e281-475c-a401-887f631f9094-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c2d5cf9-e281-475c-a401-887f631f9094-maas-api-service-ca\") pod \"4c2d5cf9-e281-475c-a401-887f631f9094\" (UID: \"4c2d5cf9-e281-475c-a401-887f631f9094\") " Nov 02 02:00:18.507016 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:18.506898 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c2d5cf9-e281-475c-a401-887f631f9094-kube-api-access-2nh8r\" (UniqueName: \"kubernetes.io/projected/4c2d5cf9-e281-475c-a401-887f631f9094-kube-api-access-2nh8r\") pod \"4c2d5cf9-e281-475c-a401-887f631f9094\" (UID: \"4c2d5cf9-e281-475c-a401-887f631f9094\") " Nov 02 02:00:18.507202 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:18.507178 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c2d5cf9-e281-475c-a401-887f631f9094-maas-api-service-ca" pod "4c2d5cf9-e281-475c-a401-887f631f9094" (UID: "4c2d5cf9-e281-475c-a401-887f631f9094"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:00:18.507685 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:18.507648 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c2d5cf9-e281-475c-a401-887f631f9094-maas-api-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:00:18.509627 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:18.509602 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2d5cf9-e281-475c-a401-887f631f9094-kube-api-access-2nh8r" pod "4c2d5cf9-e281-475c-a401-887f631f9094" (UID: "4c2d5cf9-e281-475c-a401-887f631f9094"). InnerVolumeSpecName "kube-api-access-2nh8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:00:18.608976 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:18.608949 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nh8r\" (UniqueName: \"kubernetes.io/projected/4c2d5cf9-e281-475c-a401-887f631f9094-kube-api-access-2nh8r\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:00:18.634326 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:18.634301 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f"] Nov 02 02:00:19.028788 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.028756 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:19.290250 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.290158 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" Nov 02 02:00:19.290250 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.290192 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q" event={"ID":"4c2d5cf9-e281-475c-a401-887f631f9094","Type":"ContainerDied","Data":"56173034a55e89fe55b216b2e94588862d94d944d0f2ed50b2c6f2898457105f"} Nov 02 02:00:19.290819 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.290282 5044 scope.go:122] "RemoveContainer" containerID="be0fc791352f51b2aad60d4fcc1745fc5b8e8b106f04671d53cb6c55e9911bef" Nov 02 02:00:19.299829 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.299807 5044 generic.go:356] "Generic (PLEG): container finished" podID="a2c3b60b-7def-49be-a219-34f221155180" containerID="6dda7add46aae52530f75a6cdd4f8e60941d8e71f602a48eee894f92e86af1fd" exitCode=60 Nov 02 02:00:19.299944 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.299883 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" event={"ID":"a2c3b60b-7def-49be-a219-34f221155180","Type":"ContainerDied","Data":"6dda7add46aae52530f75a6cdd4f8e60941d8e71f602a48eee894f92e86af1fd"} Nov 02 02:00:19.300335 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.300317 5044 scope.go:122] "RemoveContainer" containerID="6dda7add46aae52530f75a6cdd4f8e60941d8e71f602a48eee894f92e86af1fd" Nov 02 02:00:19.300767 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:00:19.300743 5044 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l_odh-ai-gateway-infra(a2c3b60b-7def-49be-a219-34f221155180)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" podUID="a2c3b60b-7def-49be-a219-34f221155180" Nov 02 02:00:19.304208 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.304191 5044 scope.go:122] "RemoveContainer" containerID="2e391f43da12606dc7388793f1acce945a36316b2960cd573626b8475297963e" Nov 02 02:00:19.476778 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.476749 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q"] Nov 02 02:00:19.483183 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.482531 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e06ea2-29893080-lwf8q"] Nov 02 02:00:19.556043 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.555960 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" Nov 02 02:00:19.627828 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.627439 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a10a923-11d9-4a35-a277-1ca557bdbf53-kube-api-access-789nj\" (UniqueName: \"kubernetes.io/projected/9a10a923-11d9-4a35-a277-1ca557bdbf53-kube-api-access-789nj\") pod \"9a10a923-11d9-4a35-a277-1ca557bdbf53\" (UID: \"9a10a923-11d9-4a35-a277-1ca557bdbf53\") " Nov 02 02:00:19.627828 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.627571 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a10a923-11d9-4a35-a277-1ca557bdbf53-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a10a923-11d9-4a35-a277-1ca557bdbf53-maas-api-service-ca\") pod \"9a10a923-11d9-4a35-a277-1ca557bdbf53\" (UID: \"9a10a923-11d9-4a35-a277-1ca557bdbf53\") " Nov 02 02:00:19.629461 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.629416 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a10a923-11d9-4a35-a277-1ca557bdbf53-maas-api-service-ca" pod "9a10a923-11d9-4a35-a277-1ca557bdbf53" (UID: "9a10a923-11d9-4a35-a277-1ca557bdbf53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:00:19.632905 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.632879 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a10a923-11d9-4a35-a277-1ca557bdbf53-kube-api-access-789nj" pod "9a10a923-11d9-4a35-a277-1ca557bdbf53" (UID: "9a10a923-11d9-4a35-a277-1ca557bdbf53"). InnerVolumeSpecName "kube-api-access-789nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:00:19.657895 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.657867 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" Nov 02 02:00:19.663370 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.663342 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l"] Nov 02 02:00:19.729693 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.729641 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c347ed5b-42dc-460d-97a4-bb0f386c30ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c347ed5b-42dc-460d-97a4-bb0f386c30ed-maas-api-service-ca\") pod \"c347ed5b-42dc-460d-97a4-bb0f386c30ed\" (UID: \"c347ed5b-42dc-460d-97a4-bb0f386c30ed\") " Nov 02 02:00:19.729850 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.729792 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c347ed5b-42dc-460d-97a4-bb0f386c30ed-kube-api-access-9kgg8\" (UniqueName: \"kubernetes.io/projected/c347ed5b-42dc-460d-97a4-bb0f386c30ed-kube-api-access-9kgg8\") pod \"c347ed5b-42dc-460d-97a4-bb0f386c30ed\" (UID: \"c347ed5b-42dc-460d-97a4-bb0f386c30ed\") " Nov 02 02:00:19.729991 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.729969 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c347ed5b-42dc-460d-97a4-bb0f386c30ed-maas-api-service-ca" pod "c347ed5b-42dc-460d-97a4-bb0f386c30ed" (UID: "c347ed5b-42dc-460d-97a4-bb0f386c30ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:00:19.730256 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.730233 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-789nj\" (UniqueName: \"kubernetes.io/projected/9a10a923-11d9-4a35-a277-1ca557bdbf53-kube-api-access-789nj\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:00:19.730332 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.730259 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a10a923-11d9-4a35-a277-1ca557bdbf53-maas-api-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:00:19.730332 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.730268 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c347ed5b-42dc-460d-97a4-bb0f386c30ed-maas-api-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:00:19.731879 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.731858 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c347ed5b-42dc-460d-97a4-bb0f386c30ed-kube-api-access-9kgg8" pod "c347ed5b-42dc-460d-97a4-bb0f386c30ed" (UID: "c347ed5b-42dc-460d-97a4-bb0f386c30ed"). InnerVolumeSpecName "kube-api-access-9kgg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:00:19.831185 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:19.831113 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kgg8\" (UniqueName: \"kubernetes.io/projected/c347ed5b-42dc-460d-97a4-bb0f386c30ed-kube-api-access-9kgg8\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:00:20.308150 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:20.308119 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" Nov 02 02:00:20.308644 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:20.308124 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb" event={"ID":"9a10a923-11d9-4a35-a277-1ca557bdbf53","Type":"ContainerDied","Data":"c4f9b23e83f9dca0053ad35403932b03983eaff58fee9741d0c288fb1cc25058"} Nov 02 02:00:20.308644 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:20.308265 5044 scope.go:122] "RemoveContainer" containerID="269bb8748b9c580bd3a20d9311fd6b28e0c21011f374dab2381d12d72dfecfb7" Nov 02 02:00:20.310058 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:20.310040 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" Nov 02 02:00:20.310175 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:20.310099 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f" event={"ID":"c347ed5b-42dc-460d-97a4-bb0f386c30ed","Type":"ContainerDied","Data":"43b63d65468bfd3b78e4569f462daa41d8eeaffb1ac2e7531a67b39491705d75"} Nov 02 02:00:20.321860 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:20.321839 5044 scope.go:122] "RemoveContainer" containerID="61c0a1ab9a0f7d9e1c89afa4e6ec9707bd6abacc94572982e92ab9e6e4c1560a" Nov 02 02:00:20.464615 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:20.464583 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" Nov 02 02:00:20.486539 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:20.483764 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f"] Nov 02 02:00:20.488373 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:20.488352 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-257466-29893080-wll9f"] Nov 02 02:00:20.542005 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:20.541975 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2c3b60b-7def-49be-a219-34f221155180-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c3b60b-7def-49be-a219-34f221155180-maas-api-service-ca\") pod \"a2c3b60b-7def-49be-a219-34f221155180\" (UID: \"a2c3b60b-7def-49be-a219-34f221155180\") " Nov 02 02:00:20.542206 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:20.542060 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c3b60b-7def-49be-a219-34f221155180-kube-api-access-wqp7h\" (UniqueName: \"kubernetes.io/projected/a2c3b60b-7def-49be-a219-34f221155180-kube-api-access-wqp7h\") pod \"a2c3b60b-7def-49be-a219-34f221155180\" (UID: \"a2c3b60b-7def-49be-a219-34f221155180\") " Nov 02 02:00:20.542344 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:20.542321 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c3b60b-7def-49be-a219-34f221155180-maas-api-service-ca" pod "a2c3b60b-7def-49be-a219-34f221155180" (UID: "a2c3b60b-7def-49be-a219-34f221155180"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:00:20.542728 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:20.542666 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c3b60b-7def-49be-a219-34f221155180-maas-api-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:00:20.544711 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:20.544679 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c3b60b-7def-49be-a219-34f221155180-kube-api-access-wqp7h" pod "a2c3b60b-7def-49be-a219-34f221155180" (UID: "a2c3b60b-7def-49be-a219-34f221155180"). InnerVolumeSpecName "kube-api-access-wqp7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:00:20.596887 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:20.596821 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb"] Nov 02 02:00:20.603844 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:20.603819 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5888dc-29893080-vs9hb"] Nov 02 02:00:20.645770 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:20.645729 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqp7h\" (UniqueName: \"kubernetes.io/projected/a2c3b60b-7def-49be-a219-34f221155180-kube-api-access-wqp7h\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:00:21.333967 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:21.333935 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" Nov 02 02:00:21.334454 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:21.333936 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l" event={"ID":"a2c3b60b-7def-49be-a219-34f221155180","Type":"ContainerDied","Data":"95379fdac2d6b38784bc3ab630c85821e92d38fd48d217453bac283f9da943f6"} Nov 02 02:00:21.334454 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:21.334123 5044 scope.go:122] "RemoveContainer" containerID="6dda7add46aae52530f75a6cdd4f8e60941d8e71f602a48eee894f92e86af1fd" Nov 02 02:00:21.453857 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:21.453826 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c2d5cf9-e281-475c-a401-887f631f9094" path="/var/lib/kubelet/pods/4c2d5cf9-e281-475c-a401-887f631f9094/volumes" Nov 02 02:00:21.454289 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:21.454273 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a10a923-11d9-4a35-a277-1ca557bdbf53" path="/var/lib/kubelet/pods/9a10a923-11d9-4a35-a277-1ca557bdbf53/volumes" Nov 02 02:00:21.454702 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:21.454689 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c347ed5b-42dc-460d-97a4-bb0f386c30ed" path="/var/lib/kubelet/pods/c347ed5b-42dc-460d-97a4-bb0f386c30ed/volumes" Nov 02 02:00:21.503273 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:21.503171 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l"] Nov 02 02:00:21.507304 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:21.507279 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-19436b-29893080-j624l"] Nov 02 02:00:23.453493 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:23.453458 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2c3b60b-7def-49be-a219-34f221155180" path="/var/lib/kubelet/pods/a2c3b60b-7def-49be-a219-34f221155180/volumes" Nov 02 02:00:24.176040 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:00:24.175996 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:00:24.626081 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:24.626049 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:24.849803 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:24.849774 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:24.995934 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:24.995857 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:29.999282 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:29.999250 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:35.514831 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:35.514772 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:40.874925 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:40.874897 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:45.423907 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:45.423873 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:45.965010 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:45.964971 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4"] Nov 02 02:00:45.965265 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:45.965240 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" podUID="29375ec5-a360-4efd-8a4d-9590d0337a2b" containerName="istio-proxy" containerID="cri-o://d506aba1241de5f981a53b43a231db8b5c8da42b8b323778b0c59cb3467ef5e9" gracePeriod=30 Nov 02 02:00:45.978329 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:45.978299 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:47.938050 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:47.938010 5044 patch_prober.go:29] interesting pod/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 02:00:47.938484 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:47.938088 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" podUID="29375ec5-a360-4efd-8a4d-9590d0337a2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 02:00:50.826959 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:50.826924 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:50.989352 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:50.989319 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:51.190627 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.190588 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:51.284060 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.284034 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 02:00:51.382830 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.382755 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-workload-socket\") pod \"29375ec5-a360-4efd-8a4d-9590d0337a2b\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " Nov 02 02:00:51.382830 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.382821 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-podinfo\") pod \"29375ec5-a360-4efd-8a4d-9590d0337a2b\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " Nov 02 02:00:51.383058 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.382869 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-workload-certs\") pod \"29375ec5-a360-4efd-8a4d-9590d0337a2b\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " Nov 02 02:00:51.383058 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.382910 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-token\" (UniqueName: \"kubernetes.io/projected/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-token\") pod \"29375ec5-a360-4efd-8a4d-9590d0337a2b\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " Nov 02 02:00:51.383058 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.382974 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-data\") pod \"29375ec5-a360-4efd-8a4d-9590d0337a2b\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " Nov 02 02:00:51.383058 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.383039 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-workload-socket" pod "29375ec5-a360-4efd-8a4d-9590d0337a2b" (UID: "29375ec5-a360-4efd-8a4d-9590d0337a2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:00:51.383058 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.383053 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-envoy\") pod \"29375ec5-a360-4efd-8a4d-9590d0337a2b\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " Nov 02 02:00:51.383320 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.383159 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29375ec5-a360-4efd-8a4d-9590d0337a2b-kube-api-access-lg79k\" (UniqueName: \"kubernetes.io/projected/29375ec5-a360-4efd-8a4d-9590d0337a2b-kube-api-access-lg79k\") pod \"29375ec5-a360-4efd-8a4d-9590d0337a2b\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " Nov 02 02:00:51.383320 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.383206 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-workload-certs" pod "29375ec5-a360-4efd-8a4d-9590d0337a2b" (UID: "29375ec5-a360-4efd-8a4d-9590d0337a2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:00:51.383320 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.383249 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-credential-socket\") pod \"29375ec5-a360-4efd-8a4d-9590d0337a2b\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " Nov 02 02:00:51.383320 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.383296 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29375ec5-a360-4efd-8a4d-9590d0337a2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29375ec5-a360-4efd-8a4d-9590d0337a2b-istiod-ca-cert\") pod \"29375ec5-a360-4efd-8a4d-9590d0337a2b\" (UID: \"29375ec5-a360-4efd-8a4d-9590d0337a2b\") " Nov 02 02:00:51.383588 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.383293 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-data" pod "29375ec5-a360-4efd-8a4d-9590d0337a2b" (UID: "29375ec5-a360-4efd-8a4d-9590d0337a2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:00:51.383588 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.383495 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-credential-socket" pod "29375ec5-a360-4efd-8a4d-9590d0337a2b" (UID: "29375ec5-a360-4efd-8a4d-9590d0337a2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:00:51.383743 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.383599 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29375ec5-a360-4efd-8a4d-9590d0337a2b-istiod-ca-cert" pod "29375ec5-a360-4efd-8a4d-9590d0337a2b" (UID: "29375ec5-a360-4efd-8a4d-9590d0337a2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:00:51.383993 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.383969 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:00:51.384069 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.383999 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:00:51.384069 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.384015 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:00:51.384069 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.384034 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:00:51.384069 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.384049 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29375ec5-a360-4efd-8a4d-9590d0337a2b-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:00:51.385350 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.385322 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-podinfo" pod "29375ec5-a360-4efd-8a4d-9590d0337a2b" (UID: "29375ec5-a360-4efd-8a4d-9590d0337a2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 02:00:51.385455 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.385326 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-envoy" pod "29375ec5-a360-4efd-8a4d-9590d0337a2b" (UID: "29375ec5-a360-4efd-8a4d-9590d0337a2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:00:51.385455 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.385364 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-token" pod "29375ec5-a360-4efd-8a4d-9590d0337a2b" (UID: "29375ec5-a360-4efd-8a4d-9590d0337a2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:00:51.385455 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.385422 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29375ec5-a360-4efd-8a4d-9590d0337a2b-kube-api-access-lg79k" pod "29375ec5-a360-4efd-8a4d-9590d0337a2b" (UID: "29375ec5-a360-4efd-8a4d-9590d0337a2b"). InnerVolumeSpecName "kube-api-access-lg79k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:00:51.485122 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.485087 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:00:51.485122 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.485118 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:00:51.485122 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.485128 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29375ec5-a360-4efd-8a4d-9590d0337a2b-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:00:51.485377 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.485138 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg79k\" (UniqueName: \"kubernetes.io/projected/29375ec5-a360-4efd-8a4d-9590d0337a2b-kube-api-access-lg79k\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:00:51.576750 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.576701 5044 generic.go:356] "Generic (PLEG): container finished" podID="29375ec5-a360-4efd-8a4d-9590d0337a2b" containerID="d506aba1241de5f981a53b43a231db8b5c8da42b8b323778b0c59cb3467ef5e9" exitCode=0 Nov 02 02:00:51.576952 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.576764 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" Nov 02 02:00:51.576952 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.576787 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" event={"ID":"29375ec5-a360-4efd-8a4d-9590d0337a2b","Type":"ContainerDied","Data":"d506aba1241de5f981a53b43a231db8b5c8da42b8b323778b0c59cb3467ef5e9"} Nov 02 02:00:51.576952 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.576878 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4" event={"ID":"29375ec5-a360-4efd-8a4d-9590d0337a2b","Type":"ContainerDied","Data":"ae8645a267e7baf27e39ad9098eb17a092282617334ae96d854efd49d430375b"} Nov 02 02:00:51.576952 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.576910 5044 scope.go:122] "RemoveContainer" containerID="d506aba1241de5f981a53b43a231db8b5c8da42b8b323778b0c59cb3467ef5e9" Nov 02 02:00:51.587276 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.587255 5044 scope.go:122] "RemoveContainer" containerID="d506aba1241de5f981a53b43a231db8b5c8da42b8b323778b0c59cb3467ef5e9" Nov 02 02:00:51.587562 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:00:51.587541 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d506aba1241de5f981a53b43a231db8b5c8da42b8b323778b0c59cb3467ef5e9\": container with ID starting with d506aba1241de5f981a53b43a231db8b5c8da42b8b323778b0c59cb3467ef5e9 not found: ID does not exist" containerID="d506aba1241de5f981a53b43a231db8b5c8da42b8b323778b0c59cb3467ef5e9" Nov 02 02:00:51.587613 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.587569 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d506aba1241de5f981a53b43a231db8b5c8da42b8b323778b0c59cb3467ef5e9"} err="failed to get container status \"d506aba1241de5f981a53b43a231db8b5c8da42b8b323778b0c59cb3467ef5e9\": rpc error: code = NotFound desc = could not find container \"d506aba1241de5f981a53b43a231db8b5c8da42b8b323778b0c59cb3467ef5e9\": container with ID starting with d506aba1241de5f981a53b43a231db8b5c8da42b8b323778b0c59cb3467ef5e9 not found: ID does not exist" Nov 02 02:00:51.748479 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.748393 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4"] Nov 02 02:00:51.751974 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:51.751943 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-v8lb4"] Nov 02 02:00:53.453859 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:53.453825 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29375ec5-a360-4efd-8a4d-9590d0337a2b" path="/var/lib/kubelet/pods/29375ec5-a360-4efd-8a4d-9590d0337a2b/volumes" Nov 02 02:00:54.176909 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:00:54.176867 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:00:55.789020 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:55.788986 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:56.406920 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.406884 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:56.584281 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.584212 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:00:56.825721 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.825679 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj"] Nov 02 02:00:56.827071 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827044 5044 cpu_manager.go:414] "containerMap: removing container" podUID="c347ed5b-42dc-460d-97a4-bb0f386c30ed" containerName="cleanup" Nov 02 02:00:56.827071 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827071 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c347ed5b-42dc-460d-97a4-bb0f386c30ed" containerName="cleanup" Nov 02 02:00:56.827276 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827092 5044 cpu_manager.go:414] "containerMap: removing container" podUID="9a10a923-11d9-4a35-a277-1ca557bdbf53" containerName="cleanup" Nov 02 02:00:56.827276 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827100 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a10a923-11d9-4a35-a277-1ca557bdbf53" containerName="cleanup" Nov 02 02:00:56.827276 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827109 5044 cpu_manager.go:414] "containerMap: removing container" podUID="29375ec5-a360-4efd-8a4d-9590d0337a2b" containerName="istio-proxy" Nov 02 02:00:56.827276 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827117 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29375ec5-a360-4efd-8a4d-9590d0337a2b" containerName="istio-proxy" Nov 02 02:00:56.827276 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827146 5044 cpu_manager.go:414] "containerMap: removing container" podUID="4c2d5cf9-e281-475c-a401-887f631f9094" containerName="cleanup" Nov 02 02:00:56.827276 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827154 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c2d5cf9-e281-475c-a401-887f631f9094" containerName="cleanup" Nov 02 02:00:56.827276 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827186 5044 cpu_manager.go:414] "containerMap: removing container" podUID="9a10a923-11d9-4a35-a277-1ca557bdbf53" containerName="cleanup" Nov 02 02:00:56.827276 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827193 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a10a923-11d9-4a35-a277-1ca557bdbf53" containerName="cleanup" Nov 02 02:00:56.827276 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827203 5044 cpu_manager.go:414] "containerMap: removing container" podUID="4c2d5cf9-e281-475c-a401-887f631f9094" containerName="cleanup" Nov 02 02:00:56.827276 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827210 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c2d5cf9-e281-475c-a401-887f631f9094" containerName="cleanup" Nov 02 02:00:56.827276 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827226 5044 cpu_manager.go:414] "containerMap: removing container" podUID="a2c3b60b-7def-49be-a219-34f221155180" containerName="cleanup" Nov 02 02:00:56.827276 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827234 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c3b60b-7def-49be-a219-34f221155180" containerName="cleanup" Nov 02 02:00:56.827887 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827291 5044 cpu_manager.go:414] "containerMap: removing container" podUID="a2c3b60b-7def-49be-a219-34f221155180" containerName="cleanup" Nov 02 02:00:56.827887 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827299 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c3b60b-7def-49be-a219-34f221155180" containerName="cleanup" Nov 02 02:00:56.827887 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827356 5044 cpu_manager.go:414] "containerMap: removing container" podUID="9a10a923-11d9-4a35-a277-1ca557bdbf53" containerName="cleanup" Nov 02 02:00:56.827887 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827364 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a10a923-11d9-4a35-a277-1ca557bdbf53" containerName="cleanup" Nov 02 02:00:56.827887 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827384 5044 cpu_manager.go:414] "containerMap: removing container" podUID="c347ed5b-42dc-460d-97a4-bb0f386c30ed" containerName="cleanup" Nov 02 02:00:56.827887 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827391 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c347ed5b-42dc-460d-97a4-bb0f386c30ed" containerName="cleanup" Nov 02 02:00:56.827887 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827723 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="c347ed5b-42dc-460d-97a4-bb0f386c30ed" containerName="cleanup" Nov 02 02:00:56.827887 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827747 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c3b60b-7def-49be-a219-34f221155180" containerName="cleanup" Nov 02 02:00:56.827887 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827772 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c2d5cf9-e281-475c-a401-887f631f9094" containerName="cleanup" Nov 02 02:00:56.827887 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827783 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a10a923-11d9-4a35-a277-1ca557bdbf53" containerName="cleanup" Nov 02 02:00:56.827887 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827799 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="c347ed5b-42dc-460d-97a4-bb0f386c30ed" containerName="cleanup" Nov 02 02:00:56.827887 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827824 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a10a923-11d9-4a35-a277-1ca557bdbf53" containerName="cleanup" Nov 02 02:00:56.827887 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827840 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="29375ec5-a360-4efd-8a4d-9590d0337a2b" containerName="istio-proxy" Nov 02 02:00:56.827887 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827855 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a10a923-11d9-4a35-a277-1ca557bdbf53" containerName="cleanup" Nov 02 02:00:56.827887 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827872 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c2d5cf9-e281-475c-a401-887f631f9094" containerName="cleanup" Nov 02 02:00:56.827887 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.827890 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c3b60b-7def-49be-a219-34f221155180" containerName="cleanup" Nov 02 02:00:56.839871 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.839827 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj" Nov 02 02:00:56.883034 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.883002 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj"] Nov 02 02:00:56.930456 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.930409 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-269235-dockercfg-d2xlv\"" Nov 02 02:00:56.930456 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.930419 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-269235\"" Nov 02 02:00:56.965760 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.960821 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c5bcecb-ef1c-4233-98b4-6461ff9ee181-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj\" (UID: \"4c5bcecb-ef1c-4233-98b4-6461ff9ee181\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj" Nov 02 02:00:56.965760 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.964453 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bwqw\" (UniqueName: \"kubernetes.io/projected/4c5bcecb-ef1c-4233-98b4-6461ff9ee181-kube-api-access-2bwqw\") pod \"payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj\" (UID: \"4c5bcecb-ef1c-4233-98b4-6461ff9ee181\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj" Nov 02 02:00:56.999723 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.990444 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc"] Nov 02 02:00:56.999925 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.999802 5044 cpu_manager.go:414] "containerMap: removing container" podUID="a2c3b60b-7def-49be-a219-34f221155180" containerName="cleanup" Nov 02 02:00:56.999925 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.999831 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c3b60b-7def-49be-a219-34f221155180" containerName="cleanup" Nov 02 02:00:56.999925 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.999902 5044 cpu_manager.go:414] "containerMap: removing container" podUID="c347ed5b-42dc-460d-97a4-bb0f386c30ed" containerName="cleanup" Nov 02 02:00:56.999925 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.999911 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c347ed5b-42dc-460d-97a4-bb0f386c30ed" containerName="cleanup" Nov 02 02:00:57.000152 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.999940 5044 cpu_manager.go:414] "containerMap: removing container" podUID="4c2d5cf9-e281-475c-a401-887f631f9094" containerName="cleanup" Nov 02 02:00:57.000152 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:56.999948 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c2d5cf9-e281-475c-a401-887f631f9094" containerName="cleanup" Nov 02 02:00:57.000152 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.000121 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c2d5cf9-e281-475c-a401-887f631f9094" containerName="cleanup" Nov 02 02:00:57.000152 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.000146 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c3b60b-7def-49be-a219-34f221155180" containerName="cleanup" Nov 02 02:00:57.000374 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.000156 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="c347ed5b-42dc-460d-97a4-bb0f386c30ed" containerName="cleanup" Nov 02 02:00:57.005871 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.005843 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc" Nov 02 02:00:57.041382 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.041351 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc"] Nov 02 02:00:57.065802 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.065766 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c5bcecb-ef1c-4233-98b4-6461ff9ee181-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj\" (UID: \"4c5bcecb-ef1c-4233-98b4-6461ff9ee181\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj" Nov 02 02:00:57.066047 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.065937 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bwqw\" (UniqueName: \"kubernetes.io/projected/4c5bcecb-ef1c-4233-98b4-6461ff9ee181-kube-api-access-2bwqw\") pod \"payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj\" (UID: \"4c5bcecb-ef1c-4233-98b4-6461ff9ee181\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj" Nov 02 02:00:57.066815 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.066788 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c5bcecb-ef1c-4233-98b4-6461ff9ee181-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj\" (UID: \"4c5bcecb-ef1c-4233-98b4-6461ff9ee181\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj" Nov 02 02:00:57.131133 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.131067 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bwqw\" (UniqueName: \"kubernetes.io/projected/4c5bcecb-ef1c-4233-98b4-6461ff9ee181-kube-api-access-2bwqw\") pod \"payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj\" (UID: \"4c5bcecb-ef1c-4233-98b4-6461ff9ee181\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj" Nov 02 02:00:57.159716 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.159690 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj" Nov 02 02:00:57.167007 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.166984 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj7fq\" (UniqueName: \"kubernetes.io/projected/2135d51a-e3c9-43fe-956b-f49e6a5b3efe-kube-api-access-pj7fq\") pod \"payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc\" (UID: \"2135d51a-e3c9-43fe-956b-f49e6a5b3efe\") " pod="openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc" Nov 02 02:00:57.167093 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.167020 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2135d51a-e3c9-43fe-956b-f49e6a5b3efe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc\" (UID: \"2135d51a-e3c9-43fe-956b-f49e6a5b3efe\") " pod="openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc" Nov 02 02:00:57.269193 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.269156 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj7fq\" (UniqueName: \"kubernetes.io/projected/2135d51a-e3c9-43fe-956b-f49e6a5b3efe-kube-api-access-pj7fq\") pod \"payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc\" (UID: \"2135d51a-e3c9-43fe-956b-f49e6a5b3efe\") " pod="openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc" Nov 02 02:00:57.269850 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.269209 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2135d51a-e3c9-43fe-956b-f49e6a5b3efe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc\" (UID: \"2135d51a-e3c9-43fe-956b-f49e6a5b3efe\") " pod="openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc" Nov 02 02:00:57.271186 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.271158 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2135d51a-e3c9-43fe-956b-f49e6a5b3efe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc\" (UID: \"2135d51a-e3c9-43fe-956b-f49e6a5b3efe\") " pod="openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc" Nov 02 02:00:57.334365 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.334337 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj7fq\" (UniqueName: \"kubernetes.io/projected/2135d51a-e3c9-43fe-956b-f49e6a5b3efe-kube-api-access-pj7fq\") pod \"payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc\" (UID: \"2135d51a-e3c9-43fe-956b-f49e6a5b3efe\") " pod="openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc" Nov 02 02:00:57.540186 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.540153 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj"] Nov 02 02:00:57.540995 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:00:57.540968 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5bcecb_ef1c_4233_98b4_6461ff9ee181.slice/crio-102d2f38441c379fe560a6a65a1dd6452808fc1734290a4b9003b517af750791 WatchSource:0}: Error finding container 102d2f38441c379fe560a6a65a1dd6452808fc1734290a4b9003b517af750791: Status 404 returned error can't find the container with id 102d2f38441c379fe560a6a65a1dd6452808fc1734290a4b9003b517af750791 Nov 02 02:00:57.622987 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.622955 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj" event={"ID":"4c5bcecb-ef1c-4233-98b4-6461ff9ee181","Type":"ContainerStarted","Data":"4ec6120e463c2c53ea1620fca29916adab30eb5250cafd1542ba7587f985e4ff"} Nov 02 02:00:57.623108 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.623014 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj" event={"ID":"4c5bcecb-ef1c-4233-98b4-6461ff9ee181","Type":"ContainerStarted","Data":"102d2f38441c379fe560a6a65a1dd6452808fc1734290a4b9003b517af750791"} Nov 02 02:00:57.623108 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.623041 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj" Nov 02 02:00:57.632340 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.632313 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc" Nov 02 02:00:57.744365 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:57.744300 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj" podStartSLOduration=1.744280649 podStartE2EDuration="1.744280649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:57.741344346 +0000 UTC m=+2174.412597029" watchObservedRunningTime="2026-11-02 02:00:57.744280649 +0000 UTC m=+2174.415533335" Nov 02 02:00:58.021917 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:00:58.021887 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2135d51a_e3c9_43fe_956b_f49e6a5b3efe.slice/crio-d3287d78d349751b566de54e071c83ea064de9ec7596740a9502ffc4efc57f22 WatchSource:0}: Error finding container d3287d78d349751b566de54e071c83ea064de9ec7596740a9502ffc4efc57f22: Status 404 returned error can't find the container with id d3287d78d349751b566de54e071c83ea064de9ec7596740a9502ffc4efc57f22 Nov 02 02:00:58.023867 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:58.023838 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc"] Nov 02 02:00:58.630876 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:58.630839 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc" event={"ID":"2135d51a-e3c9-43fe-956b-f49e6a5b3efe","Type":"ContainerStarted","Data":"01f81ebe3238b5a2aadb835b617da775746ba200df7eb63714b6c28c996e1332"} Nov 02 02:00:58.631090 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:58.630897 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc" event={"ID":"2135d51a-e3c9-43fe-956b-f49e6a5b3efe","Type":"ContainerStarted","Data":"d3287d78d349751b566de54e071c83ea064de9ec7596740a9502ffc4efc57f22"} Nov 02 02:00:58.631090 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:58.630924 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc" Nov 02 02:00:58.754710 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:00:58.754640 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc" podStartSLOduration=2.754627057 podStartE2EDuration="2.754627057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:00:58.752796146 +0000 UTC m=+2175.424048827" watchObservedRunningTime="2026-11-02 02:00:58.754627057 +0000 UTC m=+2175.425879767" Nov 02 02:01:00.614904 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:00.614869 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:05.473230 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:05.473176 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:05.685403 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:05.685365 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:06.220339 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:06.220306 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:08.642214 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:08.642179 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj" Nov 02 02:01:09.641447 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:09.641418 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc" Nov 02 02:01:10.156173 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.156137 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r"] Nov 02 02:01:10.156994 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.156961 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" podUID="9bcbb6c9-810a-475f-915c-5c05b23a2f3f" containerName="maas-api" containerID="cri-o://bea8318e7b4a65de8282680e42952d3103555b6755f0b276a310ef759a8acb25" gracePeriod=30 Nov 02 02:01:10.255975 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.255940 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:10.473711 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.473682 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" Nov 02 02:01:10.582691 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.569075 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-kube-api-access-fqn7v\" (UniqueName: \"kubernetes.io/projected/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-kube-api-access-fqn7v\") pod \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\" (UID: \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\") " Nov 02 02:01:10.582691 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.569247 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-openshift-service-ca\") pod \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\" (UID: \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\") " Nov 02 02:01:10.582691 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.569360 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-metrics-certs\") pod \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\" (UID: \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\") " Nov 02 02:01:10.582691 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.569394 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-maas-api-tls\") pod \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\" (UID: \"9bcbb6c9-810a-475f-915c-5c05b23a2f3f\") " Nov 02 02:01:10.582691 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.575043 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-openshift-service-ca" pod "9bcbb6c9-810a-475f-915c-5c05b23a2f3f" (UID: "9bcbb6c9-810a-475f-915c-5c05b23a2f3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:01:10.626158 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.618808 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-maas-api-tls" pod "9bcbb6c9-810a-475f-915c-5c05b23a2f3f" (UID: "9bcbb6c9-810a-475f-915c-5c05b23a2f3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:01:10.626158 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.622684 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-kube-api-access-fqn7v" pod "9bcbb6c9-810a-475f-915c-5c05b23a2f3f" (UID: "9bcbb6c9-810a-475f-915c-5c05b23a2f3f"). InnerVolumeSpecName "kube-api-access-fqn7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:01:10.626158 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.624746 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-metrics-certs" pod "9bcbb6c9-810a-475f-915c-5c05b23a2f3f" (UID: "9bcbb6c9-810a-475f-915c-5c05b23a2f3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:01:10.671751 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.671717 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqn7v\" (UniqueName: \"kubernetes.io/projected/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-kube-api-access-fqn7v\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:10.671751 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.671747 5044 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-openshift-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:10.671751 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.671758 5044 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-metrics-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:10.671981 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.671767 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bcbb6c9-810a-475f-915c-5c05b23a2f3f-maas-api-tls\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:10.738210 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.738106 5044 generic.go:356] "Generic (PLEG): container finished" podID="9bcbb6c9-810a-475f-915c-5c05b23a2f3f" containerID="bea8318e7b4a65de8282680e42952d3103555b6755f0b276a310ef759a8acb25" exitCode=0 Nov 02 02:01:10.738397 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.738324 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" event={"ID":"9bcbb6c9-810a-475f-915c-5c05b23a2f3f","Type":"ContainerDied","Data":"bea8318e7b4a65de8282680e42952d3103555b6755f0b276a310ef759a8acb25"} Nov 02 02:01:10.738397 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.738369 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" event={"ID":"9bcbb6c9-810a-475f-915c-5c05b23a2f3f","Type":"ContainerDied","Data":"fb000cf2aba01c7b6f9f124c880b68c8bd1bae990316c5613fa75c787795ef34"} Nov 02 02:01:10.738516 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.738397 5044 scope.go:122] "RemoveContainer" containerID="bea8318e7b4a65de8282680e42952d3103555b6755f0b276a310ef759a8acb25" Nov 02 02:01:10.738570 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.738548 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r" Nov 02 02:01:10.760715 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.758846 5044 scope.go:122] "RemoveContainer" containerID="bea8318e7b4a65de8282680e42952d3103555b6755f0b276a310ef759a8acb25" Nov 02 02:01:10.760715 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:01:10.759204 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bea8318e7b4a65de8282680e42952d3103555b6755f0b276a310ef759a8acb25\": container with ID starting with bea8318e7b4a65de8282680e42952d3103555b6755f0b276a310ef759a8acb25 not found: ID does not exist" containerID="bea8318e7b4a65de8282680e42952d3103555b6755f0b276a310ef759a8acb25" Nov 02 02:01:10.760715 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.759234 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea8318e7b4a65de8282680e42952d3103555b6755f0b276a310ef759a8acb25"} err="failed to get container status \"bea8318e7b4a65de8282680e42952d3103555b6755f0b276a310ef759a8acb25\": rpc error: code = NotFound desc = could not find container \"bea8318e7b4a65de8282680e42952d3103555b6755f0b276a310ef759a8acb25\": container with ID starting with bea8318e7b4a65de8282680e42952d3103555b6755f0b276a310ef759a8acb25 not found: ID does not exist" Nov 02 02:01:10.910009 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.909972 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r"] Nov 02 02:01:10.915812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:10.915787 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a7c06d-79f5879bf5-zpc8r"] Nov 02 02:01:11.460041 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:11.459537 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bcbb6c9-810a-475f-915c-5c05b23a2f3f" path="/var/lib/kubelet/pods/9bcbb6c9-810a-475f-915c-5c05b23a2f3f/volumes" Nov 02 02:01:11.496664 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:11.496630 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:11.675156 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:11.675124 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:12.073140 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.073068 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x"] Nov 02 02:01:12.074262 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.074227 5044 cpu_manager.go:414] "containerMap: removing container" podUID="9bcbb6c9-810a-475f-915c-5c05b23a2f3f" containerName="maas-api" Nov 02 02:01:12.074262 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.074260 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bcbb6c9-810a-475f-915c-5c05b23a2f3f" containerName="maas-api" Nov 02 02:01:12.074479 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.074454 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bcbb6c9-810a-475f-915c-5c05b23a2f3f" containerName="maas-api" Nov 02 02:01:12.079869 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.079842 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x" Nov 02 02:01:12.134050 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.134021 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-69c15d-dockercfg-klssd\"" Nov 02 02:01:12.134244 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.134061 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-69c15d\"" Nov 02 02:01:12.136227 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.136200 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x"] Nov 02 02:01:12.220452 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.220414 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6909a85d-37c8-4f16-907d-a893731003c3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x\" (UID: \"6909a85d-37c8-4f16-907d-a893731003c3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x" Nov 02 02:01:12.220885 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.220859 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngqwh\" (UniqueName: \"kubernetes.io/projected/6909a85d-37c8-4f16-907d-a893731003c3-kube-api-access-ngqwh\") pod \"payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x\" (UID: \"6909a85d-37c8-4f16-907d-a893731003c3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x" Nov 02 02:01:12.241098 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.241063 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v"] Nov 02 02:01:12.245581 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.245560 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v" Nov 02 02:01:12.302632 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.302596 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v"] Nov 02 02:01:12.322136 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.322106 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56409aed-4dd3-4b2a-962c-7ab8bf857fed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v\" (UID: \"56409aed-4dd3-4b2a-962c-7ab8bf857fed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v" Nov 02 02:01:12.322306 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.322161 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdfg9\" (UniqueName: \"kubernetes.io/projected/56409aed-4dd3-4b2a-962c-7ab8bf857fed-kube-api-access-pdfg9\") pod \"payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v\" (UID: \"56409aed-4dd3-4b2a-962c-7ab8bf857fed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v" Nov 02 02:01:12.322306 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.322287 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngqwh\" (UniqueName: \"kubernetes.io/projected/6909a85d-37c8-4f16-907d-a893731003c3-kube-api-access-ngqwh\") pod \"payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x\" (UID: \"6909a85d-37c8-4f16-907d-a893731003c3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x" Nov 02 02:01:12.322439 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.322425 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6909a85d-37c8-4f16-907d-a893731003c3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x\" (UID: \"6909a85d-37c8-4f16-907d-a893731003c3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x" Nov 02 02:01:12.323132 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.323111 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6909a85d-37c8-4f16-907d-a893731003c3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x\" (UID: \"6909a85d-37c8-4f16-907d-a893731003c3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x" Nov 02 02:01:12.384774 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.384703 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngqwh\" (UniqueName: \"kubernetes.io/projected/6909a85d-37c8-4f16-907d-a893731003c3-kube-api-access-ngqwh\") pod \"payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x\" (UID: \"6909a85d-37c8-4f16-907d-a893731003c3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x" Nov 02 02:01:12.390022 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.389985 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:12.397125 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.397106 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x" Nov 02 02:01:12.424080 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.424054 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56409aed-4dd3-4b2a-962c-7ab8bf857fed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v\" (UID: \"56409aed-4dd3-4b2a-962c-7ab8bf857fed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v" Nov 02 02:01:12.424215 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.424130 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdfg9\" (UniqueName: \"kubernetes.io/projected/56409aed-4dd3-4b2a-962c-7ab8bf857fed-kube-api-access-pdfg9\") pod \"payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v\" (UID: \"56409aed-4dd3-4b2a-962c-7ab8bf857fed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v" Nov 02 02:01:12.424844 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.424821 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56409aed-4dd3-4b2a-962c-7ab8bf857fed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v\" (UID: \"56409aed-4dd3-4b2a-962c-7ab8bf857fed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v" Nov 02 02:01:12.486503 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.486473 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdfg9\" (UniqueName: \"kubernetes.io/projected/56409aed-4dd3-4b2a-962c-7ab8bf857fed-kube-api-access-pdfg9\") pod \"payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v\" (UID: \"56409aed-4dd3-4b2a-962c-7ab8bf857fed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v" Nov 02 02:01:12.542476 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.542429 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:12.562422 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.562389 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v" Nov 02 02:01:12.708845 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.708814 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:12.787910 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.787880 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x"] Nov 02 02:01:12.789090 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:01:12.789060 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6909a85d_37c8_4f16_907d_a893731003c3.slice/crio-5ea27d137bdc4e46d37812980b61e0b65817a73849a8e9bd4ef82d0dd8b95309 WatchSource:0}: Error finding container 5ea27d137bdc4e46d37812980b61e0b65817a73849a8e9bd4ef82d0dd8b95309: Status 404 returned error can't find the container with id 5ea27d137bdc4e46d37812980b61e0b65817a73849a8e9bd4ef82d0dd8b95309 Nov 02 02:01:12.949305 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:12.949281 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v"] Nov 02 02:01:12.950218 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:01:12.950194 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56409aed_4dd3_4b2a_962c_7ab8bf857fed.slice/crio-7d45698d4a10c47d2ef19f62dbbf2448c6202465fa31f7b60a5fa7a5fa0b3366 WatchSource:0}: Error finding container 7d45698d4a10c47d2ef19f62dbbf2448c6202465fa31f7b60a5fa7a5fa0b3366: Status 404 returned error can't find the container with id 7d45698d4a10c47d2ef19f62dbbf2448c6202465fa31f7b60a5fa7a5fa0b3366 Nov 02 02:01:13.571320 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:13.571286 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:13.772542 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:13.772502 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x" event={"ID":"6909a85d-37c8-4f16-907d-a893731003c3","Type":"ContainerStarted","Data":"4447259a88a1835fb993e4f6c2fded9305232ccde19398c7916d309661450d95"} Nov 02 02:01:13.772756 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:13.772565 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x" event={"ID":"6909a85d-37c8-4f16-907d-a893731003c3","Type":"ContainerStarted","Data":"5ea27d137bdc4e46d37812980b61e0b65817a73849a8e9bd4ef82d0dd8b95309"} Nov 02 02:01:13.772756 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:13.772597 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x" Nov 02 02:01:13.774198 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:13.774175 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v" event={"ID":"56409aed-4dd3-4b2a-962c-7ab8bf857fed","Type":"ContainerStarted","Data":"1f9c6cf0b677f92f83d3977bcfdc3cce8cae0b2b043f500dd8e162fcc6df01ad"} Nov 02 02:01:13.774318 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:13.774217 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v" event={"ID":"56409aed-4dd3-4b2a-962c-7ab8bf857fed","Type":"ContainerStarted","Data":"7d45698d4a10c47d2ef19f62dbbf2448c6202465fa31f7b60a5fa7a5fa0b3366"} Nov 02 02:01:13.774318 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:13.774275 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v" Nov 02 02:01:13.895284 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:13.895176 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x" podStartSLOduration=1.895163115 podStartE2EDuration="1.895163115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:13.892469659 +0000 UTC m=+2190.563722343" watchObservedRunningTime="2026-11-02 02:01:13.895163115 +0000 UTC m=+2190.566415797" Nov 02 02:01:14.024739 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:14.019791 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v" podStartSLOduration=2.019772959 podStartE2EDuration="2.019772959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:01:14.018019996 +0000 UTC m=+2190.689272677" watchObservedRunningTime="2026-11-02 02:01:14.019772959 +0000 UTC m=+2190.691025640" Nov 02 02:01:15.359845 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:15.359815 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:15.529529 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:15.529040 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:18.113388 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:18.113341 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:20.021175 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:20.021142 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:20.165693 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:20.165638 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:20.371737 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:20.371625 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w"] Nov 02 02:01:20.371924 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:20.371900 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" podUID="1098cb6c-8b36-4df8-87d9-357d3ed510e0" containerName="istio-proxy" containerID="cri-o://8d0c81b411e50d6058a15e53a285c2760aa057e4b0ecee478d611e03a50cd792" gracePeriod=30 Nov 02 02:01:20.375079 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:20.375043 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:23.466712 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:23.466611 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:24.174741 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:01:24.174699 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:01:24.440981 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:24.440907 5044 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 02:01:24.440981 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:24.440966 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" podUID="1098cb6c-8b36-4df8-87d9-357d3ed510e0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 02:01:24.784584 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:24.784557 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x" Nov 02 02:01:24.785127 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:24.784622 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v" Nov 02 02:01:25.670116 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.670088 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 02:01:25.746579 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.746525 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istiod-ca-cert\") pod \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " Nov 02 02:01:25.746778 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.746591 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-workload-socket\") pod \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " Nov 02 02:01:25.746778 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.746663 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-envoy\") pod \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " Nov 02 02:01:25.746892 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.746796 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-credential-socket\") pod \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " Nov 02 02:01:25.746952 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.746882 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-workload-socket" pod "1098cb6c-8b36-4df8-87d9-357d3ed510e0" (UID: "1098cb6c-8b36-4df8-87d9-357d3ed510e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:01:25.746952 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.746908 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-data\") pod \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " Nov 02 02:01:25.747063 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.747030 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1098cb6c-8b36-4df8-87d9-357d3ed510e0-kube-api-access-7dlzj\" (UniqueName: \"kubernetes.io/projected/1098cb6c-8b36-4df8-87d9-357d3ed510e0-kube-api-access-7dlzj\") pod \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " Nov 02 02:01:25.747063 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.747047 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-credential-socket" pod "1098cb6c-8b36-4df8-87d9-357d3ed510e0" (UID: "1098cb6c-8b36-4df8-87d9-357d3ed510e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:01:25.747176 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.747068 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-podinfo\") pod \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " Nov 02 02:01:25.747176 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.747125 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-workload-certs\") pod \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " Nov 02 02:01:25.747176 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.747128 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istiod-ca-cert" pod "1098cb6c-8b36-4df8-87d9-357d3ed510e0" (UID: "1098cb6c-8b36-4df8-87d9-357d3ed510e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:01:25.747176 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.747139 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-data" pod "1098cb6c-8b36-4df8-87d9-357d3ed510e0" (UID: "1098cb6c-8b36-4df8-87d9-357d3ed510e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:01:25.747176 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.747154 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-token\" (UniqueName: \"kubernetes.io/projected/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-token\") pod \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\" (UID: \"1098cb6c-8b36-4df8-87d9-357d3ed510e0\") " Nov 02 02:01:25.747456 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.747345 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-workload-certs" pod "1098cb6c-8b36-4df8-87d9-357d3ed510e0" (UID: "1098cb6c-8b36-4df8-87d9-357d3ed510e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:01:25.747848 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.747822 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:25.747984 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.747848 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:25.747984 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.747866 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:25.747984 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.747880 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:25.747984 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.747895 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:25.749370 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.749347 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-podinfo" pod "1098cb6c-8b36-4df8-87d9-357d3ed510e0" (UID: "1098cb6c-8b36-4df8-87d9-357d3ed510e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 02:01:25.749482 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.749418 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-envoy" pod "1098cb6c-8b36-4df8-87d9-357d3ed510e0" (UID: "1098cb6c-8b36-4df8-87d9-357d3ed510e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:01:25.749608 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.749589 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1098cb6c-8b36-4df8-87d9-357d3ed510e0-kube-api-access-7dlzj" pod "1098cb6c-8b36-4df8-87d9-357d3ed510e0" (UID: "1098cb6c-8b36-4df8-87d9-357d3ed510e0"). InnerVolumeSpecName "kube-api-access-7dlzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:01:25.749783 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.749660 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-token" pod "1098cb6c-8b36-4df8-87d9-357d3ed510e0" (UID: "1098cb6c-8b36-4df8-87d9-357d3ed510e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:01:25.849563 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.849533 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dlzj\" (UniqueName: \"kubernetes.io/projected/1098cb6c-8b36-4df8-87d9-357d3ed510e0-kube-api-access-7dlzj\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:25.849563 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.849561 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:25.849563 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.849572 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:25.850056 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.849581 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1098cb6c-8b36-4df8-87d9-357d3ed510e0-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:25.870347 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.870307 5044 generic.go:356] "Generic (PLEG): container finished" podID="1098cb6c-8b36-4df8-87d9-357d3ed510e0" containerID="8d0c81b411e50d6058a15e53a285c2760aa057e4b0ecee478d611e03a50cd792" exitCode=0 Nov 02 02:01:25.870522 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.870367 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" Nov 02 02:01:25.870522 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.870396 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" event={"ID":"1098cb6c-8b36-4df8-87d9-357d3ed510e0","Type":"ContainerDied","Data":"8d0c81b411e50d6058a15e53a285c2760aa057e4b0ecee478d611e03a50cd792"} Nov 02 02:01:25.870522 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.870448 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w" event={"ID":"1098cb6c-8b36-4df8-87d9-357d3ed510e0","Type":"ContainerDied","Data":"7dd2705747b176df246fbe0921868bed9cd30258792099a60277fbf8001c2195"} Nov 02 02:01:25.870522 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.870479 5044 scope.go:122] "RemoveContainer" containerID="8d0c81b411e50d6058a15e53a285c2760aa057e4b0ecee478d611e03a50cd792" Nov 02 02:01:25.882778 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.882753 5044 scope.go:122] "RemoveContainer" containerID="f9a7355bb7e05b8ad35aaeaf18aab7ec02f1ca3c2d163e0837e5d253a6305359" Nov 02 02:01:25.892913 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.892893 5044 scope.go:122] "RemoveContainer" containerID="8d0c81b411e50d6058a15e53a285c2760aa057e4b0ecee478d611e03a50cd792" Nov 02 02:01:25.893197 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:01:25.893172 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0c81b411e50d6058a15e53a285c2760aa057e4b0ecee478d611e03a50cd792\": container with ID starting with 8d0c81b411e50d6058a15e53a285c2760aa057e4b0ecee478d611e03a50cd792 not found: ID does not exist" containerID="8d0c81b411e50d6058a15e53a285c2760aa057e4b0ecee478d611e03a50cd792" Nov 02 02:01:25.893277 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.893208 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0c81b411e50d6058a15e53a285c2760aa057e4b0ecee478d611e03a50cd792"} err="failed to get container status \"8d0c81b411e50d6058a15e53a285c2760aa057e4b0ecee478d611e03a50cd792\": rpc error: code = NotFound desc = could not find container \"8d0c81b411e50d6058a15e53a285c2760aa057e4b0ecee478d611e03a50cd792\": container with ID starting with 8d0c81b411e50d6058a15e53a285c2760aa057e4b0ecee478d611e03a50cd792 not found: ID does not exist" Nov 02 02:01:25.893277 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.893231 5044 scope.go:122] "RemoveContainer" containerID="f9a7355bb7e05b8ad35aaeaf18aab7ec02f1ca3c2d163e0837e5d253a6305359" Nov 02 02:01:25.893482 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:01:25.893462 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a7355bb7e05b8ad35aaeaf18aab7ec02f1ca3c2d163e0837e5d253a6305359\": container with ID starting with f9a7355bb7e05b8ad35aaeaf18aab7ec02f1ca3c2d163e0837e5d253a6305359 not found: ID does not exist" containerID="f9a7355bb7e05b8ad35aaeaf18aab7ec02f1ca3c2d163e0837e5d253a6305359" Nov 02 02:01:25.893542 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:25.893488 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a7355bb7e05b8ad35aaeaf18aab7ec02f1ca3c2d163e0837e5d253a6305359"} err="failed to get container status \"f9a7355bb7e05b8ad35aaeaf18aab7ec02f1ca3c2d163e0837e5d253a6305359\": rpc error: code = NotFound desc = could not find container \"f9a7355bb7e05b8ad35aaeaf18aab7ec02f1ca3c2d163e0837e5d253a6305359\": container with ID starting with f9a7355bb7e05b8ad35aaeaf18aab7ec02f1ca3c2d163e0837e5d253a6305359 not found: ID does not exist" Nov 02 02:01:26.041334 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:26.041305 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w"] Nov 02 02:01:26.044765 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:26.044731 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a7c06d-openshift-default-85947547bf-9zl7w"] Nov 02 02:01:27.454289 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:27.454252 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1098cb6c-8b36-4df8-87d9-357d3ed510e0" path="/var/lib/kubelet/pods/1098cb6c-8b36-4df8-87d9-357d3ed510e0/volumes" Nov 02 02:01:28.661093 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:28.661052 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:32.648833 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:32.648801 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:32.792821 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:32.792782 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:33.202273 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:33.202219 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:33.784071 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:33.784032 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:33.933736 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:33.933705 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:34.416490 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:34.416457 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:34.565029 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:34.564995 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:35.267622 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:35.267579 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:36.957827 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:36.957792 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:37.235560 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:37.235466 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:37.882434 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:37.882371 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:40.388278 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:40.388232 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:44.127871 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:44.127833 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:44.283142 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:44.283109 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:44.460663 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:44.460581 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:44.856039 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:44.856009 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf"] Nov 02 02:01:44.856226 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:44.856187 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" podUID="e0072c75-d48f-4a49-a1ba-5586e673fd36" containerName="maas-api" containerID="cri-o://cad05d7289ad4df0895604e7298868435eb1378ef610bcd473703e8b6086edf4" gracePeriod=30 Nov 02 02:01:44.965977 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:44.965911 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:45.013972 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:45.013936 5044 generic.go:356] "Generic (PLEG): container finished" podID="e0072c75-d48f-4a49-a1ba-5586e673fd36" containerID="cad05d7289ad4df0895604e7298868435eb1378ef610bcd473703e8b6086edf4" exitCode=0 Nov 02 02:01:45.014156 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:45.014006 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" event={"ID":"e0072c75-d48f-4a49-a1ba-5586e673fd36","Type":"ContainerDied","Data":"cad05d7289ad4df0895604e7298868435eb1378ef610bcd473703e8b6086edf4"} Nov 02 02:01:45.174508 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:45.174476 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" Nov 02 02:01:45.226578 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:45.226548 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0072c75-d48f-4a49-a1ba-5586e673fd36-kube-api-access-w5ffv\" (UniqueName: \"kubernetes.io/projected/e0072c75-d48f-4a49-a1ba-5586e673fd36-kube-api-access-w5ffv\") pod \"e0072c75-d48f-4a49-a1ba-5586e673fd36\" (UID: \"e0072c75-d48f-4a49-a1ba-5586e673fd36\") " Nov 02 02:01:45.226731 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:45.226635 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0072c75-d48f-4a49-a1ba-5586e673fd36-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0072c75-d48f-4a49-a1ba-5586e673fd36-openshift-service-ca\") pod \"e0072c75-d48f-4a49-a1ba-5586e673fd36\" (UID: \"e0072c75-d48f-4a49-a1ba-5586e673fd36\") " Nov 02 02:01:45.226794 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:45.226746 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0072c75-d48f-4a49-a1ba-5586e673fd36-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0072c75-d48f-4a49-a1ba-5586e673fd36-maas-api-tls\") pod \"e0072c75-d48f-4a49-a1ba-5586e673fd36\" (UID: \"e0072c75-d48f-4a49-a1ba-5586e673fd36\") " Nov 02 02:01:45.226832 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:45.226822 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0072c75-d48f-4a49-a1ba-5586e673fd36-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0072c75-d48f-4a49-a1ba-5586e673fd36-metrics-certs\") pod \"e0072c75-d48f-4a49-a1ba-5586e673fd36\" (UID: \"e0072c75-d48f-4a49-a1ba-5586e673fd36\") " Nov 02 02:01:45.227057 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:45.227031 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0072c75-d48f-4a49-a1ba-5586e673fd36-openshift-service-ca" pod "e0072c75-d48f-4a49-a1ba-5586e673fd36" (UID: "e0072c75-d48f-4a49-a1ba-5586e673fd36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:01:45.227261 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:45.227244 5044 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0072c75-d48f-4a49-a1ba-5586e673fd36-openshift-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:45.228844 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:45.228817 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0072c75-d48f-4a49-a1ba-5586e673fd36-metrics-certs" pod "e0072c75-d48f-4a49-a1ba-5586e673fd36" (UID: "e0072c75-d48f-4a49-a1ba-5586e673fd36"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:01:45.228957 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:45.228921 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0072c75-d48f-4a49-a1ba-5586e673fd36-kube-api-access-w5ffv" pod "e0072c75-d48f-4a49-a1ba-5586e673fd36" (UID: "e0072c75-d48f-4a49-a1ba-5586e673fd36"). InnerVolumeSpecName "kube-api-access-w5ffv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:01:45.229059 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:45.229044 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0072c75-d48f-4a49-a1ba-5586e673fd36-maas-api-tls" pod "e0072c75-d48f-4a49-a1ba-5586e673fd36" (UID: "e0072c75-d48f-4a49-a1ba-5586e673fd36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:01:45.363259 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:45.333774 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0072c75-d48f-4a49-a1ba-5586e673fd36-maas-api-tls\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:45.363259 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:45.333811 5044 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0072c75-d48f-4a49-a1ba-5586e673fd36-metrics-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:45.363259 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:45.333827 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5ffv\" (UniqueName: \"kubernetes.io/projected/e0072c75-d48f-4a49-a1ba-5586e673fd36-kube-api-access-w5ffv\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:45.774220 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:45.774190 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:46.022656 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.022625 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" Nov 02 02:01:46.022656 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.022661 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf" event={"ID":"e0072c75-d48f-4a49-a1ba-5586e673fd36","Type":"ContainerDied","Data":"edd59f30c65eb685e34e52d6476fea5306a7743a391c6d94e771929feff0fe7e"} Nov 02 02:01:46.022926 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.022737 5044 scope.go:122] "RemoveContainer" containerID="cad05d7289ad4df0895604e7298868435eb1378ef610bcd473703e8b6086edf4" Nov 02 02:01:46.119270 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.119229 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4"] Nov 02 02:01:46.119495 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.119471 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" podUID="66b4726e-ea0f-46e5-b1a8-cf803f344ef4" containerName="maas-api" containerID="cri-o://d50fa6a11b74a0ce2308f765cd5b94f65d52fcfddaa41e3ce5df232226499c32" gracePeriod=30 Nov 02 02:01:46.206021 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.205977 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf"] Nov 02 02:01:46.211121 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.211091 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-109b07-588c754c4b-mrdxf"] Nov 02 02:01:46.238050 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.238025 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:46.454118 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.454093 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" Nov 02 02:01:46.582745 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.578908 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-maas-api-tls\") pod \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\" (UID: \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\") " Nov 02 02:01:46.582745 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.579017 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-metrics-certs\") pod \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\" (UID: \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\") " Nov 02 02:01:46.582745 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.579109 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-kube-api-access-lc8j5\" (UniqueName: \"kubernetes.io/projected/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-kube-api-access-lc8j5\") pod \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\" (UID: \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\") " Nov 02 02:01:46.582745 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.579281 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-openshift-service-ca\") pod \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\" (UID: \"66b4726e-ea0f-46e5-b1a8-cf803f344ef4\") " Nov 02 02:01:46.588700 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.585977 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-openshift-service-ca" pod "66b4726e-ea0f-46e5-b1a8-cf803f344ef4" (UID: "66b4726e-ea0f-46e5-b1a8-cf803f344ef4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:01:46.592704 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.591212 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-kube-api-access-lc8j5" pod "66b4726e-ea0f-46e5-b1a8-cf803f344ef4" (UID: "66b4726e-ea0f-46e5-b1a8-cf803f344ef4"). InnerVolumeSpecName "kube-api-access-lc8j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:01:46.596687 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.594002 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:46.596687 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.596102 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-maas-api-tls" pod "66b4726e-ea0f-46e5-b1a8-cf803f344ef4" (UID: "66b4726e-ea0f-46e5-b1a8-cf803f344ef4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:01:46.601981 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.601949 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-metrics-certs" pod "66b4726e-ea0f-46e5-b1a8-cf803f344ef4" (UID: "66b4726e-ea0f-46e5-b1a8-cf803f344ef4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:01:46.681537 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.681504 5044 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-metrics-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:46.681537 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.681535 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc8j5\" (UniqueName: \"kubernetes.io/projected/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-kube-api-access-lc8j5\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:46.681772 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.681545 5044 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-openshift-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:46.681772 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:46.681553 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66b4726e-ea0f-46e5-b1a8-cf803f344ef4-maas-api-tls\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:01:47.031161 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:47.031124 5044 generic.go:356] "Generic (PLEG): container finished" podID="66b4726e-ea0f-46e5-b1a8-cf803f344ef4" containerID="d50fa6a11b74a0ce2308f765cd5b94f65d52fcfddaa41e3ce5df232226499c32" exitCode=0 Nov 02 02:01:47.031331 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:47.031172 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" Nov 02 02:01:47.031331 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:47.031209 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" event={"ID":"66b4726e-ea0f-46e5-b1a8-cf803f344ef4","Type":"ContainerDied","Data":"d50fa6a11b74a0ce2308f765cd5b94f65d52fcfddaa41e3ce5df232226499c32"} Nov 02 02:01:47.031331 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:47.031261 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4" event={"ID":"66b4726e-ea0f-46e5-b1a8-cf803f344ef4","Type":"ContainerDied","Data":"a071bd1857c6712db764f9081508993068f9e0e2ed6ccc2b7f89da9c072f8798"} Nov 02 02:01:47.031331 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:47.031290 5044 scope.go:122] "RemoveContainer" containerID="d50fa6a11b74a0ce2308f765cd5b94f65d52fcfddaa41e3ce5df232226499c32" Nov 02 02:01:47.033869 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:47.033848 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:47.042037 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:47.042021 5044 scope.go:122] "RemoveContainer" containerID="d50fa6a11b74a0ce2308f765cd5b94f65d52fcfddaa41e3ce5df232226499c32" Nov 02 02:01:47.042305 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:01:47.042287 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d50fa6a11b74a0ce2308f765cd5b94f65d52fcfddaa41e3ce5df232226499c32\": container with ID starting with d50fa6a11b74a0ce2308f765cd5b94f65d52fcfddaa41e3ce5df232226499c32 not found: ID does not exist" containerID="d50fa6a11b74a0ce2308f765cd5b94f65d52fcfddaa41e3ce5df232226499c32" Nov 02 02:01:47.042354 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:47.042312 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d50fa6a11b74a0ce2308f765cd5b94f65d52fcfddaa41e3ce5df232226499c32"} err="failed to get container status \"d50fa6a11b74a0ce2308f765cd5b94f65d52fcfddaa41e3ce5df232226499c32\": rpc error: code = NotFound desc = could not find container \"d50fa6a11b74a0ce2308f765cd5b94f65d52fcfddaa41e3ce5df232226499c32\": container with ID starting with d50fa6a11b74a0ce2308f765cd5b94f65d52fcfddaa41e3ce5df232226499c32 not found: ID does not exist" Nov 02 02:01:47.217765 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:47.217722 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4"] Nov 02 02:01:47.223433 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:47.223391 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-9w4n4"] Nov 02 02:01:47.455478 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:47.455387 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b4726e-ea0f-46e5-b1a8-cf803f344ef4" path="/var/lib/kubelet/pods/66b4726e-ea0f-46e5-b1a8-cf803f344ef4/volumes" Nov 02 02:01:47.455969 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:47.455946 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0072c75-d48f-4a49-a1ba-5586e673fd36" path="/var/lib/kubelet/pods/e0072c75-d48f-4a49-a1ba-5586e673fd36/volumes" Nov 02 02:01:47.666945 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:47.666901 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:48.102120 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:48.102078 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:48.247429 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:48.247389 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:48.608505 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:48.608309 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:52.191177 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:52.191144 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:53.369402 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:53.369366 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:53.856761 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:53.856731 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:54.019584 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:54.019550 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:54.203740 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:01:54.201650 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:01:54.238207 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:54.238173 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:55.765865 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:55.765828 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:55.933022 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:55.932975 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:56.098806 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:56.098728 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:01:58.944341 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:01:58.944303 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:00.118153 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:00.118119 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg"] Nov 02 02:02:00.230199 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:00.230164 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff"] Nov 02 02:02:00.482043 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:00.481963 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:00.627703 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:00.627655 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:01.161080 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:01.161040 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h"] Nov 02 02:02:01.167160 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:01.167136 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj"] Nov 02 02:02:01.178006 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:01.177981 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5"] Nov 02 02:02:01.318413 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:01.318379 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:03.881125 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:03.881090 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:05.118935 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:05.118889 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg" podUID="fcdbd493-2074-4976-aef4-9eb91f6f7522" containerName="payload-processing" containerID="cri-o://9f64fce5aaead46ed4a390609f7ab42002345f413dfe17bcf2bf60ebae942e21" gracePeriod=25 Nov 02 02:02:05.231413 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:05.231352 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff" podUID="8ee58b67-0d93-4541-b43d-0c8f790ecc38" containerName="payload-pre-processing" containerID="cri-o://750282f0f64facdac6be428db50e024cdab4d119601a10f0d75d99958eb9456f" gracePeriod=25 Nov 02 02:02:05.473782 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:05.473751 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg" Nov 02 02:02:05.537022 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:05.536994 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff" Nov 02 02:02:05.659632 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:05.659596 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee58b67-0d93-4541-b43d-0c8f790ecc38-kube-api-access-sc7m4\" (UniqueName: \"kubernetes.io/projected/8ee58b67-0d93-4541-b43d-0c8f790ecc38-kube-api-access-sc7m4\") pod \"8ee58b67-0d93-4541-b43d-0c8f790ecc38\" (UID: \"8ee58b67-0d93-4541-b43d-0c8f790ecc38\") " Nov 02 02:02:05.659866 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:05.659744 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcdbd493-2074-4976-aef4-9eb91f6f7522-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcdbd493-2074-4976-aef4-9eb91f6f7522-plugins-config-volume\") pod \"fcdbd493-2074-4976-aef4-9eb91f6f7522\" (UID: \"fcdbd493-2074-4976-aef4-9eb91f6f7522\") " Nov 02 02:02:05.659866 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:05.659816 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcdbd493-2074-4976-aef4-9eb91f6f7522-kube-api-access-l4krj\" (UniqueName: \"kubernetes.io/projected/fcdbd493-2074-4976-aef4-9eb91f6f7522-kube-api-access-l4krj\") pod \"fcdbd493-2074-4976-aef4-9eb91f6f7522\" (UID: \"fcdbd493-2074-4976-aef4-9eb91f6f7522\") " Nov 02 02:02:05.660001 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:05.659962 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ee58b67-0d93-4541-b43d-0c8f790ecc38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee58b67-0d93-4541-b43d-0c8f790ecc38-plugins-config-volume\") pod \"8ee58b67-0d93-4541-b43d-0c8f790ecc38\" (UID: \"8ee58b67-0d93-4541-b43d-0c8f790ecc38\") " Nov 02 02:02:05.660281 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:05.660251 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcdbd493-2074-4976-aef4-9eb91f6f7522-plugins-config-volume" pod "fcdbd493-2074-4976-aef4-9eb91f6f7522" (UID: "fcdbd493-2074-4976-aef4-9eb91f6f7522"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:02:05.660400 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:05.660312 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee58b67-0d93-4541-b43d-0c8f790ecc38-plugins-config-volume" pod "8ee58b67-0d93-4541-b43d-0c8f790ecc38" (UID: "8ee58b67-0d93-4541-b43d-0c8f790ecc38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:02:05.660517 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:05.660420 5044 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ee58b67-0d93-4541-b43d-0c8f790ecc38-plugins-config-volume\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:05.660517 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:05.660433 5044 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcdbd493-2074-4976-aef4-9eb91f6f7522-plugins-config-volume\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:05.661942 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:05.661916 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcdbd493-2074-4976-aef4-9eb91f6f7522-kube-api-access-l4krj" pod "fcdbd493-2074-4976-aef4-9eb91f6f7522" (UID: "fcdbd493-2074-4976-aef4-9eb91f6f7522"). InnerVolumeSpecName "kube-api-access-l4krj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:02:05.662051 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:05.662027 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee58b67-0d93-4541-b43d-0c8f790ecc38-kube-api-access-sc7m4" pod "8ee58b67-0d93-4541-b43d-0c8f790ecc38" (UID: "8ee58b67-0d93-4541-b43d-0c8f790ecc38"). InnerVolumeSpecName "kube-api-access-sc7m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:02:05.761351 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:05.761317 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc7m4\" (UniqueName: \"kubernetes.io/projected/8ee58b67-0d93-4541-b43d-0c8f790ecc38-kube-api-access-sc7m4\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:05.761351 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:05.761350 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4krj\" (UniqueName: \"kubernetes.io/projected/fcdbd493-2074-4976-aef4-9eb91f6f7522-kube-api-access-l4krj\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:06.189173 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:06.189132 5044 generic.go:356] "Generic (PLEG): container finished" podID="fcdbd493-2074-4976-aef4-9eb91f6f7522" containerID="9f64fce5aaead46ed4a390609f7ab42002345f413dfe17bcf2bf60ebae942e21" exitCode=0 Nov 02 02:02:06.189729 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:06.189205 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg" Nov 02 02:02:06.189729 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:06.189234 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg" event={"ID":"fcdbd493-2074-4976-aef4-9eb91f6f7522","Type":"ContainerDied","Data":"9f64fce5aaead46ed4a390609f7ab42002345f413dfe17bcf2bf60ebae942e21"} Nov 02 02:02:06.189729 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:06.189330 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg" event={"ID":"fcdbd493-2074-4976-aef4-9eb91f6f7522","Type":"ContainerDied","Data":"3937ab087b8cc5db311da77bf7d47a2a5a3d31c7798f2c375b3e2eae2f727173"} Nov 02 02:02:06.189729 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:06.189361 5044 scope.go:122] "RemoveContainer" containerID="9f64fce5aaead46ed4a390609f7ab42002345f413dfe17bcf2bf60ebae942e21" Nov 02 02:02:06.191845 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:06.191812 5044 generic.go:356] "Generic (PLEG): container finished" podID="8ee58b67-0d93-4541-b43d-0c8f790ecc38" containerID="750282f0f64facdac6be428db50e024cdab4d119601a10f0d75d99958eb9456f" exitCode=0 Nov 02 02:02:06.191947 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:06.191914 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff" event={"ID":"8ee58b67-0d93-4541-b43d-0c8f790ecc38","Type":"ContainerDied","Data":"750282f0f64facdac6be428db50e024cdab4d119601a10f0d75d99958eb9456f"} Nov 02 02:02:06.192027 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:06.191960 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff" event={"ID":"8ee58b67-0d93-4541-b43d-0c8f790ecc38","Type":"ContainerDied","Data":"077bb70d1b0bb5d0cb9fa04745e7e5366e93f7c74c2010bf1a48ef9b6583109d"} Nov 02 02:02:06.192147 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:06.192042 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff" Nov 02 02:02:06.206579 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:06.206561 5044 scope.go:122] "RemoveContainer" containerID="9f64fce5aaead46ed4a390609f7ab42002345f413dfe17bcf2bf60ebae942e21" Nov 02 02:02:06.207005 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:02:06.206983 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f64fce5aaead46ed4a390609f7ab42002345f413dfe17bcf2bf60ebae942e21\": container with ID starting with 9f64fce5aaead46ed4a390609f7ab42002345f413dfe17bcf2bf60ebae942e21 not found: ID does not exist" containerID="9f64fce5aaead46ed4a390609f7ab42002345f413dfe17bcf2bf60ebae942e21" Nov 02 02:02:06.207089 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:06.207016 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f64fce5aaead46ed4a390609f7ab42002345f413dfe17bcf2bf60ebae942e21"} err="failed to get container status \"9f64fce5aaead46ed4a390609f7ab42002345f413dfe17bcf2bf60ebae942e21\": rpc error: code = NotFound desc = could not find container \"9f64fce5aaead46ed4a390609f7ab42002345f413dfe17bcf2bf60ebae942e21\": container with ID starting with 9f64fce5aaead46ed4a390609f7ab42002345f413dfe17bcf2bf60ebae942e21 not found: ID does not exist" Nov 02 02:02:06.207089 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:06.207034 5044 scope.go:122] "RemoveContainer" containerID="750282f0f64facdac6be428db50e024cdab4d119601a10f0d75d99958eb9456f" Nov 02 02:02:06.216491 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:06.216462 5044 scope.go:122] "RemoveContainer" containerID="750282f0f64facdac6be428db50e024cdab4d119601a10f0d75d99958eb9456f" Nov 02 02:02:06.216757 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:02:06.216739 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750282f0f64facdac6be428db50e024cdab4d119601a10f0d75d99958eb9456f\": container with ID starting with 750282f0f64facdac6be428db50e024cdab4d119601a10f0d75d99958eb9456f not found: ID does not exist" containerID="750282f0f64facdac6be428db50e024cdab4d119601a10f0d75d99958eb9456f" Nov 02 02:02:06.216821 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:06.216765 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750282f0f64facdac6be428db50e024cdab4d119601a10f0d75d99958eb9456f"} err="failed to get container status \"750282f0f64facdac6be428db50e024cdab4d119601a10f0d75d99958eb9456f\": rpc error: code = NotFound desc = could not find container \"750282f0f64facdac6be428db50e024cdab4d119601a10f0d75d99958eb9456f\": container with ID starting with 750282f0f64facdac6be428db50e024cdab4d119601a10f0d75d99958eb9456f not found: ID does not exist" Nov 02 02:02:06.360070 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:06.360028 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg"] Nov 02 02:02:06.367684 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:06.367649 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-d700e9-6d4fd99cf9-krvxg"] Nov 02 02:02:06.479330 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:06.479239 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff"] Nov 02 02:02:06.482782 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:06.482754 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d700e9-d5fc59487-zdjff"] Nov 02 02:02:07.453683 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:07.453639 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee58b67-0d93-4541-b43d-0c8f790ecc38" path="/var/lib/kubelet/pods/8ee58b67-0d93-4541-b43d-0c8f790ecc38/volumes" Nov 02 02:02:07.454135 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:07.454113 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcdbd493-2074-4976-aef4-9eb91f6f7522" path="/var/lib/kubelet/pods/fcdbd493-2074-4976-aef4-9eb91f6f7522/volumes" Nov 02 02:02:08.930865 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:08.930831 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:09.421201 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:09.421171 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t"] Nov 02 02:02:09.672343 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:09.672262 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:09.821518 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:09.821481 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:11.383775 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:11.383740 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:11.530964 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:11.530931 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:13.776817 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:13.776785 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:14.421751 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:14.421711 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t" podUID="6b201831-e055-4763-9400-bb68d825dc3e" containerName="payload-pre-processing" containerID="cri-o://0f5eb9ffef9a671465a144df12943bf6c4164d979907217d195101583dbfbf83" gracePeriod=25 Nov 02 02:02:16.186547 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.186491 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" containerID="cri-o://4d053eb250622dd6af25bc2eacebd4fd9f0b6fb873da60b1d2750a1318e3db10" gracePeriod=45 Nov 02 02:02:16.190884 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.190853 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" containerID="cri-o://d09c6b30ab2c39f70d1a48ed9474712b5f4699522d52ff746e00fa5426c34787" gracePeriod=45 Nov 02 02:02:16.203987 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.203955 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" containerID="cri-o://3c1b871342495601c96c323ca4ecaa78b0341a4d166261e399a9b3263f50689b" gracePeriod=45 Nov 02 02:02:16.460179 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.460097 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt"] Nov 02 02:02:16.460610 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.460582 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" podUID="54c547c6-30fd-43be-ac27-299da9065a25" containerName="istio-proxy" containerID="cri-o://9dac6e5a3e1ed2c1fe6d79d550399d27f0b1f199cb175e1645552506c7f32ad9" gracePeriod=30 Nov 02 02:02:16.540269 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.540247 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 02:02:16.543278 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.543253 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 02:02:16.550715 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.550658 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 02:02:16.626463 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.626425 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-home\" (UniqueName: \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-home\") pod \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " Nov 02 02:02:16.626655 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.626474 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-dshm\") pod \"cada31e5-25a0-464e-9e89-f857de57beba\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " Nov 02 02:02:16.626655 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.626509 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-home\" (UniqueName: \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-home\") pod \"cada31e5-25a0-464e-9e89-f857de57beba\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " Nov 02 02:02:16.626655 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.626542 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-dshm\") pod \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " Nov 02 02:02:16.626655 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.626572 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-tmp-dir\") pod \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " Nov 02 02:02:16.626655 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.626624 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-model-cache\") pod \"cada31e5-25a0-464e-9e89-f857de57beba\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " Nov 02 02:02:16.627629 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.626665 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-home\" (UniqueName: \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-home\") pod \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " Nov 02 02:02:16.627629 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.626801 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-kube-api-access-cpk76\" (UniqueName: \"kubernetes.io/projected/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-kube-api-access-cpk76\") pod \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " Nov 02 02:02:16.627629 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.626834 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-dshm\") pod \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " Nov 02 02:02:16.627629 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.626868 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-tls-certs\") pod \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " Nov 02 02:02:16.627629 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.626961 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cada31e5-25a0-464e-9e89-f857de57beba-kube-api-access-fvf7s\" (UniqueName: \"kubernetes.io/projected/cada31e5-25a0-464e-9e89-f857de57beba-kube-api-access-fvf7s\") pod \"cada31e5-25a0-464e-9e89-f857de57beba\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " Nov 02 02:02:16.627629 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.627005 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-tmp-dir\") pod \"cada31e5-25a0-464e-9e89-f857de57beba\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " Nov 02 02:02:16.627629 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.627060 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cada31e5-25a0-464e-9e89-f857de57beba-tls-certs\" (UniqueName: \"kubernetes.io/secret/cada31e5-25a0-464e-9e89-f857de57beba-tls-certs\") pod \"cada31e5-25a0-464e-9e89-f857de57beba\" (UID: \"cada31e5-25a0-464e-9e89-f857de57beba\") " Nov 02 02:02:16.627629 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.627116 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-home" pod "7ad28d87-78de-4b69-9f2a-4d713cbac5ad" (UID: "7ad28d87-78de-4b69-9f2a-4d713cbac5ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:02:16.627629 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.627150 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-tls-certs\") pod \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " Nov 02 02:02:16.627629 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.627185 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-tmp-dir" pod "7ad28d87-78de-4b69-9f2a-4d713cbac5ad" (UID: "7ad28d87-78de-4b69-9f2a-4d713cbac5ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:02:16.627629 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.627209 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-home" pod "a4e6f149-8398-4eb5-8fcc-f658c6e3300c" (UID: "a4e6f149-8398-4eb5-8fcc-f658c6e3300c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:02:16.627629 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.627272 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-model-cache\") pod \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " Nov 02 02:02:16.627629 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.627355 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-model-cache" pod "cada31e5-25a0-464e-9e89-f857de57beba" (UID: "cada31e5-25a0-464e-9e89-f857de57beba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:02:16.627629 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.627374 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-home" pod "cada31e5-25a0-464e-9e89-f857de57beba" (UID: "cada31e5-25a0-464e-9e89-f857de57beba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:02:16.627629 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.627427 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-tmp-dir" pod "cada31e5-25a0-464e-9e89-f857de57beba" (UID: "cada31e5-25a0-464e-9e89-f857de57beba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:02:16.628493 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.627695 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-model-cache\") pod \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\" (UID: \"7ad28d87-78de-4b69-9f2a-4d713cbac5ad\") " Nov 02 02:02:16.628493 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.627756 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-kube-api-access-dwxwt\" (UniqueName: \"kubernetes.io/projected/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-kube-api-access-dwxwt\") pod \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " Nov 02 02:02:16.628493 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.627811 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-tmp-dir\") pod \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\" (UID: \"a4e6f149-8398-4eb5-8fcc-f658c6e3300c\") " Nov 02 02:02:16.628493 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.628474 5044 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-home\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:16.628746 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.628498 5044 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-home\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:16.628746 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.628513 5044 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-tmp-dir\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:16.628746 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.628530 5044 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-model-cache\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:16.628746 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.628544 5044 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-home\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:16.628746 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.628558 5044 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-tmp-dir\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:16.629018 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.628820 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-tmp-dir" pod "a4e6f149-8398-4eb5-8fcc-f658c6e3300c" (UID: "a4e6f149-8398-4eb5-8fcc-f658c6e3300c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:02:16.629018 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.628997 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-model-cache" pod "a4e6f149-8398-4eb5-8fcc-f658c6e3300c" (UID: "a4e6f149-8398-4eb5-8fcc-f658c6e3300c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:02:16.629206 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.629168 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-model-cache" pod "7ad28d87-78de-4b69-9f2a-4d713cbac5ad" (UID: "7ad28d87-78de-4b69-9f2a-4d713cbac5ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:02:16.631261 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.631222 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cada31e5-25a0-464e-9e89-f857de57beba-tls-certs" pod "cada31e5-25a0-464e-9e89-f857de57beba" (UID: "cada31e5-25a0-464e-9e89-f857de57beba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:02:16.631642 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.631618 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-dshm" pod "7ad28d87-78de-4b69-9f2a-4d713cbac5ad" (UID: "7ad28d87-78de-4b69-9f2a-4d713cbac5ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:02:16.631893 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.631873 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-dshm" pod "cada31e5-25a0-464e-9e89-f857de57beba" (UID: "cada31e5-25a0-464e-9e89-f857de57beba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:02:16.632041 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.632013 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-kube-api-access-cpk76" pod "7ad28d87-78de-4b69-9f2a-4d713cbac5ad" (UID: "7ad28d87-78de-4b69-9f2a-4d713cbac5ad"). InnerVolumeSpecName "kube-api-access-cpk76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:02:16.632041 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.632008 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-tls-certs" pod "a4e6f149-8398-4eb5-8fcc-f658c6e3300c" (UID: "a4e6f149-8398-4eb5-8fcc-f658c6e3300c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:02:16.632172 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.632074 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-tls-certs" pod "7ad28d87-78de-4b69-9f2a-4d713cbac5ad" (UID: "7ad28d87-78de-4b69-9f2a-4d713cbac5ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:02:16.632605 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.632576 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cada31e5-25a0-464e-9e89-f857de57beba-kube-api-access-fvf7s" pod "cada31e5-25a0-464e-9e89-f857de57beba" (UID: "cada31e5-25a0-464e-9e89-f857de57beba"). InnerVolumeSpecName "kube-api-access-fvf7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:02:16.633153 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.633130 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-dshm" pod "a4e6f149-8398-4eb5-8fcc-f658c6e3300c" (UID: "a4e6f149-8398-4eb5-8fcc-f658c6e3300c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:02:16.633925 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.633903 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-kube-api-access-dwxwt" pod "a4e6f149-8398-4eb5-8fcc-f658c6e3300c" (UID: "a4e6f149-8398-4eb5-8fcc-f658c6e3300c"). InnerVolumeSpecName "kube-api-access-dwxwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:02:16.729935 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.729850 5044 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-model-cache\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:16.729935 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.729886 5044 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-model-cache\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:16.729935 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.729903 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwxwt\" (UniqueName: \"kubernetes.io/projected/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-kube-api-access-dwxwt\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:16.729935 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.729918 5044 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-tmp-dir\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:16.729935 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.729933 5044 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cada31e5-25a0-464e-9e89-f857de57beba-dshm\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:16.730242 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.729946 5044 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-dshm\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:16.730242 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.729960 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpk76\" (UniqueName: \"kubernetes.io/projected/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-kube-api-access-cpk76\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:16.730242 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.729973 5044 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-dshm\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:16.730242 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.729987 5044 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad28d87-78de-4b69-9f2a-4d713cbac5ad-tls-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:16.730242 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.730000 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvf7s\" (UniqueName: \"kubernetes.io/projected/cada31e5-25a0-464e-9e89-f857de57beba-kube-api-access-fvf7s\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:16.730242 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.730013 5044 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cada31e5-25a0-464e-9e89-f857de57beba-tls-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:16.730242 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:16.730026 5044 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e6f149-8398-4eb5-8fcc-f658c6e3300c-tls-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:17.280600 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.280568 5044 generic.go:356] "Generic (PLEG): container finished" podID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerID="4d053eb250622dd6af25bc2eacebd4fd9f0b6fb873da60b1d2750a1318e3db10" exitCode=0 Nov 02 02:02:17.281086 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.280615 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" Nov 02 02:02:17.281086 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.280651 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" event={"ID":"7ad28d87-78de-4b69-9f2a-4d713cbac5ad","Type":"ContainerDied","Data":"4d053eb250622dd6af25bc2eacebd4fd9f0b6fb873da60b1d2750a1318e3db10"} Nov 02 02:02:17.281086 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.280724 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h" event={"ID":"7ad28d87-78de-4b69-9f2a-4d713cbac5ad","Type":"ContainerDied","Data":"1bdcc4e0819773369eb6ebde099ca01c932744dd9b5306b6aac68e7d72ddf00e"} Nov 02 02:02:17.281086 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.280761 5044 scope.go:122] "RemoveContainer" containerID="4d053eb250622dd6af25bc2eacebd4fd9f0b6fb873da60b1d2750a1318e3db10" Nov 02 02:02:17.287714 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.287645 5044 generic.go:356] "Generic (PLEG): container finished" podID="cada31e5-25a0-464e-9e89-f857de57beba" containerID="d09c6b30ab2c39f70d1a48ed9474712b5f4699522d52ff746e00fa5426c34787" exitCode=0 Nov 02 02:02:17.287869 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.287853 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" Nov 02 02:02:17.287965 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.287944 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" event={"ID":"cada31e5-25a0-464e-9e89-f857de57beba","Type":"ContainerDied","Data":"d09c6b30ab2c39f70d1a48ed9474712b5f4699522d52ff746e00fa5426c34787"} Nov 02 02:02:17.288017 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.287986 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj" event={"ID":"cada31e5-25a0-464e-9e89-f857de57beba","Type":"ContainerDied","Data":"d148ee7b116c804c71c73aba4c2f6237a6d2bb10363f62e60b3d442b1b0d64aa"} Nov 02 02:02:17.290170 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.290144 5044 generic.go:356] "Generic (PLEG): container finished" podID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerID="3c1b871342495601c96c323ca4ecaa78b0341a4d166261e399a9b3263f50689b" exitCode=0 Nov 02 02:02:17.290170 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.290205 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" event={"ID":"a4e6f149-8398-4eb5-8fcc-f658c6e3300c","Type":"ContainerDied","Data":"3c1b871342495601c96c323ca4ecaa78b0341a4d166261e399a9b3263f50689b"} Nov 02 02:02:17.290429 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.290285 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" event={"ID":"a4e6f149-8398-4eb5-8fcc-f658c6e3300c","Type":"ContainerDied","Data":"bdd73960cf179e7aa2119bbdb7277bd33c507fc32ddf87efadfb1b06d693a8e9"} Nov 02 02:02:17.290573 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.290550 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" Nov 02 02:02:17.298767 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.298746 5044 scope.go:122] "RemoveContainer" containerID="4d053eb250622dd6af25bc2eacebd4fd9f0b6fb873da60b1d2750a1318e3db10" Nov 02 02:02:17.299320 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:02:17.299122 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d053eb250622dd6af25bc2eacebd4fd9f0b6fb873da60b1d2750a1318e3db10\": container with ID starting with 4d053eb250622dd6af25bc2eacebd4fd9f0b6fb873da60b1d2750a1318e3db10 not found: ID does not exist" containerID="4d053eb250622dd6af25bc2eacebd4fd9f0b6fb873da60b1d2750a1318e3db10" Nov 02 02:02:17.299320 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.299156 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d053eb250622dd6af25bc2eacebd4fd9f0b6fb873da60b1d2750a1318e3db10"} err="failed to get container status \"4d053eb250622dd6af25bc2eacebd4fd9f0b6fb873da60b1d2750a1318e3db10\": rpc error: code = NotFound desc = could not find container \"4d053eb250622dd6af25bc2eacebd4fd9f0b6fb873da60b1d2750a1318e3db10\": container with ID starting with 4d053eb250622dd6af25bc2eacebd4fd9f0b6fb873da60b1d2750a1318e3db10 not found: ID does not exist" Nov 02 02:02:17.299320 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.299181 5044 scope.go:122] "RemoveContainer" containerID="d09c6b30ab2c39f70d1a48ed9474712b5f4699522d52ff746e00fa5426c34787" Nov 02 02:02:17.309278 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.309260 5044 scope.go:122] "RemoveContainer" containerID="d09c6b30ab2c39f70d1a48ed9474712b5f4699522d52ff746e00fa5426c34787" Nov 02 02:02:17.309538 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:02:17.309519 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d09c6b30ab2c39f70d1a48ed9474712b5f4699522d52ff746e00fa5426c34787\": container with ID starting with d09c6b30ab2c39f70d1a48ed9474712b5f4699522d52ff746e00fa5426c34787 not found: ID does not exist" containerID="d09c6b30ab2c39f70d1a48ed9474712b5f4699522d52ff746e00fa5426c34787" Nov 02 02:02:17.309590 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.309545 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09c6b30ab2c39f70d1a48ed9474712b5f4699522d52ff746e00fa5426c34787"} err="failed to get container status \"d09c6b30ab2c39f70d1a48ed9474712b5f4699522d52ff746e00fa5426c34787\": rpc error: code = NotFound desc = could not find container \"d09c6b30ab2c39f70d1a48ed9474712b5f4699522d52ff746e00fa5426c34787\": container with ID starting with d09c6b30ab2c39f70d1a48ed9474712b5f4699522d52ff746e00fa5426c34787 not found: ID does not exist" Nov 02 02:02:17.309590 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.309564 5044 scope.go:122] "RemoveContainer" containerID="3c1b871342495601c96c323ca4ecaa78b0341a4d166261e399a9b3263f50689b" Nov 02 02:02:17.318844 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.318827 5044 scope.go:122] "RemoveContainer" containerID="3c1b871342495601c96c323ca4ecaa78b0341a4d166261e399a9b3263f50689b" Nov 02 02:02:17.319111 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:02:17.319087 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c1b871342495601c96c323ca4ecaa78b0341a4d166261e399a9b3263f50689b\": container with ID starting with 3c1b871342495601c96c323ca4ecaa78b0341a4d166261e399a9b3263f50689b not found: ID does not exist" containerID="3c1b871342495601c96c323ca4ecaa78b0341a4d166261e399a9b3263f50689b" Nov 02 02:02:17.319180 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.319116 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c1b871342495601c96c323ca4ecaa78b0341a4d166261e399a9b3263f50689b"} err="failed to get container status \"3c1b871342495601c96c323ca4ecaa78b0341a4d166261e399a9b3263f50689b\": rpc error: code = NotFound desc = could not find container \"3c1b871342495601c96c323ca4ecaa78b0341a4d166261e399a9b3263f50689b\": container with ID starting with 3c1b871342495601c96c323ca4ecaa78b0341a4d166261e399a9b3263f50689b not found: ID does not exist" Nov 02 02:02:17.453791 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.453759 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5"] Nov 02 02:02:17.458466 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.458413 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5"] Nov 02 02:02:17.515145 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.515107 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b71ba5/simulator-w0-b71ba5-kserve-7f8d5c7d9c-p2mg5" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 02:02:17.567915 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.567888 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h"] Nov 02 02:02:17.570939 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.570919 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/distinct-w0-b71ba5-kserve-86946c569f-z469h"] Nov 02 02:02:17.682827 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.682791 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj"] Nov 02 02:02:17.687280 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.687248 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b71ba5/premium-w0-b71ba5-kserve-5594fffdd8-nx7tj"] Nov 02 02:02:17.841765 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.841734 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct"] Nov 02 02:02:17.841936 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.841918 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" podUID="a83df443-031e-4a2e-a4f4-5a686ad12085" containerName="maas-api" containerID="cri-o://1826b0770551eaf7962311e087b7947c340cb1a7b0d69a561c88bffab28f801c" gracePeriod=30 Nov 02 02:02:17.946008 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:17.945975 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:18.167688 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.167650 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" Nov 02 02:02:18.195206 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.195179 5044 patch_prober.go:29] interesting pod/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 02:02:18.195341 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.195230 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" podUID="54c547c6-30fd-43be-ac27-299da9065a25" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 02:02:18.253694 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.252004 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83df443-031e-4a2e-a4f4-5a686ad12085-kube-api-access-5hv87\" (UniqueName: \"kubernetes.io/projected/a83df443-031e-4a2e-a4f4-5a686ad12085-kube-api-access-5hv87\") pod \"a83df443-031e-4a2e-a4f4-5a686ad12085\" (UID: \"a83df443-031e-4a2e-a4f4-5a686ad12085\") " Nov 02 02:02:18.253694 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.252544 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83df443-031e-4a2e-a4f4-5a686ad12085-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83df443-031e-4a2e-a4f4-5a686ad12085-openshift-service-ca\") pod \"a83df443-031e-4a2e-a4f4-5a686ad12085\" (UID: \"a83df443-031e-4a2e-a4f4-5a686ad12085\") " Nov 02 02:02:18.253694 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.252719 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a83df443-031e-4a2e-a4f4-5a686ad12085-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83df443-031e-4a2e-a4f4-5a686ad12085-metrics-certs\") pod \"a83df443-031e-4a2e-a4f4-5a686ad12085\" (UID: \"a83df443-031e-4a2e-a4f4-5a686ad12085\") " Nov 02 02:02:18.253694 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.252823 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a83df443-031e-4a2e-a4f4-5a686ad12085-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83df443-031e-4a2e-a4f4-5a686ad12085-maas-api-tls\") pod \"a83df443-031e-4a2e-a4f4-5a686ad12085\" (UID: \"a83df443-031e-4a2e-a4f4-5a686ad12085\") " Nov 02 02:02:18.254132 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.254023 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83df443-031e-4a2e-a4f4-5a686ad12085-openshift-service-ca" pod "a83df443-031e-4a2e-a4f4-5a686ad12085" (UID: "a83df443-031e-4a2e-a4f4-5a686ad12085"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:02:18.262691 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.259991 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83df443-031e-4a2e-a4f4-5a686ad12085-maas-api-tls" pod "a83df443-031e-4a2e-a4f4-5a686ad12085" (UID: "a83df443-031e-4a2e-a4f4-5a686ad12085"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:02:18.262691 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.261718 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83df443-031e-4a2e-a4f4-5a686ad12085-metrics-certs" pod "a83df443-031e-4a2e-a4f4-5a686ad12085" (UID: "a83df443-031e-4a2e-a4f4-5a686ad12085"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:02:18.262691 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.261769 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83df443-031e-4a2e-a4f4-5a686ad12085-kube-api-access-5hv87" pod "a83df443-031e-4a2e-a4f4-5a686ad12085" (UID: "a83df443-031e-4a2e-a4f4-5a686ad12085"). InnerVolumeSpecName "kube-api-access-5hv87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:02:18.306737 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.305653 5044 generic.go:356] "Generic (PLEG): container finished" podID="a83df443-031e-4a2e-a4f4-5a686ad12085" containerID="1826b0770551eaf7962311e087b7947c340cb1a7b0d69a561c88bffab28f801c" exitCode=0 Nov 02 02:02:18.306737 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.305880 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" event={"ID":"a83df443-031e-4a2e-a4f4-5a686ad12085","Type":"ContainerDied","Data":"1826b0770551eaf7962311e087b7947c340cb1a7b0d69a561c88bffab28f801c"} Nov 02 02:02:18.306737 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.305924 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" event={"ID":"a83df443-031e-4a2e-a4f4-5a686ad12085","Type":"ContainerDied","Data":"936db0206058a44babe29a8dcf574ec8ce8971dddb251051e2f730dc8430b248"} Nov 02 02:02:18.306737 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.305950 5044 scope.go:122] "RemoveContainer" containerID="1826b0770551eaf7962311e087b7947c340cb1a7b0d69a561c88bffab28f801c" Nov 02 02:02:18.306737 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.306040 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct" Nov 02 02:02:18.323773 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.323749 5044 scope.go:122] "RemoveContainer" containerID="1826b0770551eaf7962311e087b7947c340cb1a7b0d69a561c88bffab28f801c" Nov 02 02:02:18.324102 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:02:18.324083 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1826b0770551eaf7962311e087b7947c340cb1a7b0d69a561c88bffab28f801c\": container with ID starting with 1826b0770551eaf7962311e087b7947c340cb1a7b0d69a561c88bffab28f801c not found: ID does not exist" containerID="1826b0770551eaf7962311e087b7947c340cb1a7b0d69a561c88bffab28f801c" Nov 02 02:02:18.324184 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.324107 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1826b0770551eaf7962311e087b7947c340cb1a7b0d69a561c88bffab28f801c"} err="failed to get container status \"1826b0770551eaf7962311e087b7947c340cb1a7b0d69a561c88bffab28f801c\": rpc error: code = NotFound desc = could not find container \"1826b0770551eaf7962311e087b7947c340cb1a7b0d69a561c88bffab28f801c\": container with ID starting with 1826b0770551eaf7962311e087b7947c340cb1a7b0d69a561c88bffab28f801c not found: ID does not exist" Nov 02 02:02:18.354586 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.354556 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83df443-031e-4a2e-a4f4-5a686ad12085-maas-api-tls\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:18.354586 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.354583 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hv87\" (UniqueName: \"kubernetes.io/projected/a83df443-031e-4a2e-a4f4-5a686ad12085-kube-api-access-5hv87\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:18.354774 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.354601 5044 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83df443-031e-4a2e-a4f4-5a686ad12085-openshift-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:18.354774 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.354610 5044 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83df443-031e-4a2e-a4f4-5a686ad12085-metrics-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:18.485464 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.485378 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct"] Nov 02 02:02:18.488342 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:18.488309 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-d700e9-c879897cf-9dhct"] Nov 02 02:02:19.298079 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:19.298037 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t" podUID="6b201831-e055-4763-9400-bb68d825dc3e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 02:02:19.453211 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:19.453175 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" path="/var/lib/kubelet/pods/7ad28d87-78de-4b69-9f2a-4d713cbac5ad/volumes" Nov 02 02:02:19.453844 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:19.453825 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" path="/var/lib/kubelet/pods/a4e6f149-8398-4eb5-8fcc-f658c6e3300c/volumes" Nov 02 02:02:19.454403 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:19.454388 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a83df443-031e-4a2e-a4f4-5a686ad12085" path="/var/lib/kubelet/pods/a83df443-031e-4a2e-a4f4-5a686ad12085/volumes" Nov 02 02:02:19.455370 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:19.455353 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cada31e5-25a0-464e-9e89-f857de57beba" path="/var/lib/kubelet/pods/cada31e5-25a0-464e-9e89-f857de57beba/volumes" Nov 02 02:02:19.459183 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:19.459159 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:19.693865 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:19.693785 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:19.845201 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:19.845175 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:20.424564 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:20.424508 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:21.765003 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.764978 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 02:02:21.903550 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.903509 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-workload-socket\") pod \"54c547c6-30fd-43be-ac27-299da9065a25\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " Nov 02 02:02:21.903835 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.903576 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-istio-envoy\") pod \"54c547c6-30fd-43be-ac27-299da9065a25\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " Nov 02 02:02:21.903835 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.903624 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-credential-socket\") pod \"54c547c6-30fd-43be-ac27-299da9065a25\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " Nov 02 02:02:21.903835 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.903772 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c547c6-30fd-43be-ac27-299da9065a25-istio-token\" (UniqueName: \"kubernetes.io/projected/54c547c6-30fd-43be-ac27-299da9065a25-istio-token\") pod \"54c547c6-30fd-43be-ac27-299da9065a25\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " Nov 02 02:02:21.904029 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.903848 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c547c6-30fd-43be-ac27-299da9065a25-kube-api-access-rbm26\" (UniqueName: \"kubernetes.io/projected/54c547c6-30fd-43be-ac27-299da9065a25-kube-api-access-rbm26\") pod \"54c547c6-30fd-43be-ac27-299da9065a25\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " Nov 02 02:02:21.904029 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.903916 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-workload-socket" pod "54c547c6-30fd-43be-ac27-299da9065a25" (UID: "54c547c6-30fd-43be-ac27-299da9065a25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:02:21.904029 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.903955 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54c547c6-30fd-43be-ac27-299da9065a25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c547c6-30fd-43be-ac27-299da9065a25-istio-podinfo\") pod \"54c547c6-30fd-43be-ac27-299da9065a25\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " Nov 02 02:02:21.904029 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.903996 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-credential-socket" pod "54c547c6-30fd-43be-ac27-299da9065a25" (UID: "54c547c6-30fd-43be-ac27-299da9065a25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:02:21.904261 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.904101 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-workload-certs\") pod \"54c547c6-30fd-43be-ac27-299da9065a25\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " Nov 02 02:02:21.904261 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.904177 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54c547c6-30fd-43be-ac27-299da9065a25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c547c6-30fd-43be-ac27-299da9065a25-istiod-ca-cert\") pod \"54c547c6-30fd-43be-ac27-299da9065a25\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " Nov 02 02:02:21.904261 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.904218 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-istio-data\") pod \"54c547c6-30fd-43be-ac27-299da9065a25\" (UID: \"54c547c6-30fd-43be-ac27-299da9065a25\") " Nov 02 02:02:21.904427 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.904342 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-workload-certs" pod "54c547c6-30fd-43be-ac27-299da9065a25" (UID: "54c547c6-30fd-43be-ac27-299da9065a25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:02:21.904636 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.904616 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c547c6-30fd-43be-ac27-299da9065a25-istiod-ca-cert" pod "54c547c6-30fd-43be-ac27-299da9065a25" (UID: "54c547c6-30fd-43be-ac27-299da9065a25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:02:21.904884 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.904863 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-istio-data" pod "54c547c6-30fd-43be-ac27-299da9065a25" (UID: "54c547c6-30fd-43be-ac27-299da9065a25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:02:21.905028 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.905007 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:21.905028 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.905029 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:21.905178 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.905038 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:21.905178 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.905047 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c547c6-30fd-43be-ac27-299da9065a25-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:21.905178 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.905057 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:21.906270 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.906243 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54c547c6-30fd-43be-ac27-299da9065a25-istio-podinfo" pod "54c547c6-30fd-43be-ac27-299da9065a25" (UID: "54c547c6-30fd-43be-ac27-299da9065a25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 02:02:21.906548 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.906517 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-istio-envoy" pod "54c547c6-30fd-43be-ac27-299da9065a25" (UID: "54c547c6-30fd-43be-ac27-299da9065a25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:02:21.906548 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.906530 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c547c6-30fd-43be-ac27-299da9065a25-kube-api-access-rbm26" pod "54c547c6-30fd-43be-ac27-299da9065a25" (UID: "54c547c6-30fd-43be-ac27-299da9065a25"). InnerVolumeSpecName "kube-api-access-rbm26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:02:21.906742 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:21.906566 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c547c6-30fd-43be-ac27-299da9065a25-istio-token" pod "54c547c6-30fd-43be-ac27-299da9065a25" (UID: "54c547c6-30fd-43be-ac27-299da9065a25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:02:22.006882 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:22.006850 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c547c6-30fd-43be-ac27-299da9065a25-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:22.006882 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:22.006878 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c547c6-30fd-43be-ac27-299da9065a25-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:22.006882 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:22.006888 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbm26\" (UniqueName: \"kubernetes.io/projected/54c547c6-30fd-43be-ac27-299da9065a25-kube-api-access-rbm26\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:22.007102 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:22.006898 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c547c6-30fd-43be-ac27-299da9065a25-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:22.347977 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:22.347944 5044 generic.go:356] "Generic (PLEG): container finished" podID="54c547c6-30fd-43be-ac27-299da9065a25" containerID="9dac6e5a3e1ed2c1fe6d79d550399d27f0b1f199cb175e1645552506c7f32ad9" exitCode=0 Nov 02 02:02:22.348148 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:22.347996 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" Nov 02 02:02:22.348148 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:22.348018 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" event={"ID":"54c547c6-30fd-43be-ac27-299da9065a25","Type":"ContainerDied","Data":"9dac6e5a3e1ed2c1fe6d79d550399d27f0b1f199cb175e1645552506c7f32ad9"} Nov 02 02:02:22.348148 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:22.348088 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt" event={"ID":"54c547c6-30fd-43be-ac27-299da9065a25","Type":"ContainerDied","Data":"8a01b8a79550fe724e3e05646cef68d71f65e7a4ee6b9733142ca010b5324465"} Nov 02 02:02:22.348148 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:22.348118 5044 scope.go:122] "RemoveContainer" containerID="9dac6e5a3e1ed2c1fe6d79d550399d27f0b1f199cb175e1645552506c7f32ad9" Nov 02 02:02:22.359143 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:22.359126 5044 scope.go:122] "RemoveContainer" containerID="9dac6e5a3e1ed2c1fe6d79d550399d27f0b1f199cb175e1645552506c7f32ad9" Nov 02 02:02:22.359389 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:02:22.359371 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dac6e5a3e1ed2c1fe6d79d550399d27f0b1f199cb175e1645552506c7f32ad9\": container with ID starting with 9dac6e5a3e1ed2c1fe6d79d550399d27f0b1f199cb175e1645552506c7f32ad9 not found: ID does not exist" containerID="9dac6e5a3e1ed2c1fe6d79d550399d27f0b1f199cb175e1645552506c7f32ad9" Nov 02 02:02:22.359437 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:22.359396 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dac6e5a3e1ed2c1fe6d79d550399d27f0b1f199cb175e1645552506c7f32ad9"} err="failed to get container status \"9dac6e5a3e1ed2c1fe6d79d550399d27f0b1f199cb175e1645552506c7f32ad9\": rpc error: code = NotFound desc = could not find container \"9dac6e5a3e1ed2c1fe6d79d550399d27f0b1f199cb175e1645552506c7f32ad9\": container with ID starting with 9dac6e5a3e1ed2c1fe6d79d550399d27f0b1f199cb175e1645552506c7f32ad9 not found: ID does not exist" Nov 02 02:02:22.523871 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:22.523840 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt"] Nov 02 02:02:22.575129 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:22.575097 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-8prkt"] Nov 02 02:02:23.452411 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:23.452377 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54c547c6-30fd-43be-ac27-299da9065a25" path="/var/lib/kubelet/pods/54c547c6-30fd-43be-ac27-299da9065a25/volumes" Nov 02 02:02:24.173183 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:02:24.173147 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:02:24.805081 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:24.805052 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:27.167972 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:27.167921 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:27.309463 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:27.309407 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:27.513022 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:27.512985 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:28.261618 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:28.261588 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:28.413876 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:28.413841 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:29.134854 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:29.134820 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd"] Nov 02 02:02:29.135055 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:29.135033 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" podUID="e68f185a-890d-445d-9e1e-b35d7d0e2448" containerName="maas-api" containerID="cri-o://75b02b3181d3161f137f276f95923835a6a3bae0ebbb271424665214d3a80445" gracePeriod=30 Nov 02 02:02:29.230856 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:29.230822 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:29.299310 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:29.299273 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t" podUID="6b201831-e055-4763-9400-bb68d825dc3e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 02:02:29.425895 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:29.425865 5044 generic.go:356] "Generic (PLEG): container finished" podID="e68f185a-890d-445d-9e1e-b35d7d0e2448" containerID="75b02b3181d3161f137f276f95923835a6a3bae0ebbb271424665214d3a80445" exitCode=0 Nov 02 02:02:29.426013 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:29.425937 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" event={"ID":"e68f185a-890d-445d-9e1e-b35d7d0e2448","Type":"ContainerDied","Data":"75b02b3181d3161f137f276f95923835a6a3bae0ebbb271424665214d3a80445"} Nov 02 02:02:29.445817 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:29.445790 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" Nov 02 02:02:29.501849 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:29.501821 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68f185a-890d-445d-9e1e-b35d7d0e2448-kube-api-access-9ql9n\" (UniqueName: \"kubernetes.io/projected/e68f185a-890d-445d-9e1e-b35d7d0e2448-kube-api-access-9ql9n\") pod \"e68f185a-890d-445d-9e1e-b35d7d0e2448\" (UID: \"e68f185a-890d-445d-9e1e-b35d7d0e2448\") " Nov 02 02:02:29.502036 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:29.501928 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e68f185a-890d-445d-9e1e-b35d7d0e2448-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68f185a-890d-445d-9e1e-b35d7d0e2448-maas-api-tls\") pod \"e68f185a-890d-445d-9e1e-b35d7d0e2448\" (UID: \"e68f185a-890d-445d-9e1e-b35d7d0e2448\") " Nov 02 02:02:29.502036 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:29.501990 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e68f185a-890d-445d-9e1e-b35d7d0e2448-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68f185a-890d-445d-9e1e-b35d7d0e2448-openshift-service-ca\") pod \"e68f185a-890d-445d-9e1e-b35d7d0e2448\" (UID: \"e68f185a-890d-445d-9e1e-b35d7d0e2448\") " Nov 02 02:02:29.502164 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:29.502096 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e68f185a-890d-445d-9e1e-b35d7d0e2448-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68f185a-890d-445d-9e1e-b35d7d0e2448-metrics-certs\") pod \"e68f185a-890d-445d-9e1e-b35d7d0e2448\" (UID: \"e68f185a-890d-445d-9e1e-b35d7d0e2448\") " Nov 02 02:02:29.502617 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:29.502586 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68f185a-890d-445d-9e1e-b35d7d0e2448-openshift-service-ca" pod "e68f185a-890d-445d-9e1e-b35d7d0e2448" (UID: "e68f185a-890d-445d-9e1e-b35d7d0e2448"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:02:29.504384 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:29.504359 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68f185a-890d-445d-9e1e-b35d7d0e2448-maas-api-tls" pod "e68f185a-890d-445d-9e1e-b35d7d0e2448" (UID: "e68f185a-890d-445d-9e1e-b35d7d0e2448"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:02:29.505431 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:29.505407 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68f185a-890d-445d-9e1e-b35d7d0e2448-metrics-certs" pod "e68f185a-890d-445d-9e1e-b35d7d0e2448" (UID: "e68f185a-890d-445d-9e1e-b35d7d0e2448"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:02:29.505955 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:29.505934 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68f185a-890d-445d-9e1e-b35d7d0e2448-kube-api-access-9ql9n" pod "e68f185a-890d-445d-9e1e-b35d7d0e2448" (UID: "e68f185a-890d-445d-9e1e-b35d7d0e2448"). InnerVolumeSpecName "kube-api-access-9ql9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:02:29.606778 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:29.606730 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ql9n\" (UniqueName: \"kubernetes.io/projected/e68f185a-890d-445d-9e1e-b35d7d0e2448-kube-api-access-9ql9n\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:29.606778 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:29.606774 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68f185a-890d-445d-9e1e-b35d7d0e2448-maas-api-tls\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:29.606990 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:29.606790 5044 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68f185a-890d-445d-9e1e-b35d7d0e2448-openshift-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:29.606990 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:29.606805 5044 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68f185a-890d-445d-9e1e-b35d7d0e2448-metrics-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:30.433975 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:30.433945 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" Nov 02 02:02:30.433975 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:30.433970 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd" event={"ID":"e68f185a-890d-445d-9e1e-b35d7d0e2448","Type":"ContainerDied","Data":"179fca984674e4790b7948285834751e9e0634fe4a4ab6bc8a10918893aa8daa"} Nov 02 02:02:30.434444 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:30.434033 5044 scope.go:122] "RemoveContainer" containerID="75b02b3181d3161f137f276f95923835a6a3bae0ebbb271424665214d3a80445" Nov 02 02:02:30.604728 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:30.604688 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd"] Nov 02 02:02:30.653762 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:30.653729 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-218802-86d84859c6-jqmzd"] Nov 02 02:02:30.951783 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:30.951737 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:31.151632 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:31.151597 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:31.307541 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:31.301345 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:31.452911 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:31.452881 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e68f185a-890d-445d-9e1e-b35d7d0e2448" path="/var/lib/kubelet/pods/e68f185a-890d-445d-9e1e-b35d7d0e2448/volumes" Nov 02 02:02:31.896547 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:31.896512 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:34.708348 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:34.708316 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t" Nov 02 02:02:34.772474 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:34.772443 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b201831-e055-4763-9400-bb68d825dc3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b201831-e055-4763-9400-bb68d825dc3e-plugins-config-volume\") pod \"6b201831-e055-4763-9400-bb68d825dc3e\" (UID: \"6b201831-e055-4763-9400-bb68d825dc3e\") " Nov 02 02:02:34.772704 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:34.772644 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b201831-e055-4763-9400-bb68d825dc3e-kube-api-access-6d9zq\" (UniqueName: \"kubernetes.io/projected/6b201831-e055-4763-9400-bb68d825dc3e-kube-api-access-6d9zq\") pod \"6b201831-e055-4763-9400-bb68d825dc3e\" (UID: \"6b201831-e055-4763-9400-bb68d825dc3e\") " Nov 02 02:02:34.772874 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:34.772853 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b201831-e055-4763-9400-bb68d825dc3e-plugins-config-volume" pod "6b201831-e055-4763-9400-bb68d825dc3e" (UID: "6b201831-e055-4763-9400-bb68d825dc3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:02:34.773183 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:34.773166 5044 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b201831-e055-4763-9400-bb68d825dc3e-plugins-config-volume\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:34.775173 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:34.775145 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b201831-e055-4763-9400-bb68d825dc3e-kube-api-access-6d9zq" pod "6b201831-e055-4763-9400-bb68d825dc3e" (UID: "6b201831-e055-4763-9400-bb68d825dc3e"). InnerVolumeSpecName "kube-api-access-6d9zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:02:34.874065 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:34.874033 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d9zq\" (UniqueName: \"kubernetes.io/projected/6b201831-e055-4763-9400-bb68d825dc3e-kube-api-access-6d9zq\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:35.465277 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:35.465246 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x"] Nov 02 02:02:35.470180 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:35.470151 5044 generic.go:356] "Generic (PLEG): container finished" podID="6b201831-e055-4763-9400-bb68d825dc3e" containerID="0f5eb9ffef9a671465a144df12943bf6c4164d979907217d195101583dbfbf83" exitCode=0 Nov 02 02:02:35.470312 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:35.470206 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t" Nov 02 02:02:35.470312 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:35.470243 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t" event={"ID":"6b201831-e055-4763-9400-bb68d825dc3e","Type":"ContainerDied","Data":"0f5eb9ffef9a671465a144df12943bf6c4164d979907217d195101583dbfbf83"} Nov 02 02:02:35.470312 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:35.470296 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t" event={"ID":"6b201831-e055-4763-9400-bb68d825dc3e","Type":"ContainerDied","Data":"0026b2ed5ab73a8bda36a284bb9e33d6f0f64b2cd3e69b51ecbda42898383bb4"} Nov 02 02:02:35.470411 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:35.470322 5044 scope.go:122] "RemoveContainer" containerID="0f5eb9ffef9a671465a144df12943bf6c4164d979907217d195101583dbfbf83" Nov 02 02:02:35.482880 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:35.482859 5044 scope.go:122] "RemoveContainer" containerID="0f5eb9ffef9a671465a144df12943bf6c4164d979907217d195101583dbfbf83" Nov 02 02:02:35.483210 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:02:35.483189 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f5eb9ffef9a671465a144df12943bf6c4164d979907217d195101583dbfbf83\": container with ID starting with 0f5eb9ffef9a671465a144df12943bf6c4164d979907217d195101583dbfbf83 not found: ID does not exist" containerID="0f5eb9ffef9a671465a144df12943bf6c4164d979907217d195101583dbfbf83" Nov 02 02:02:35.483263 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:35.483218 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5eb9ffef9a671465a144df12943bf6c4164d979907217d195101583dbfbf83"} err="failed to get container status \"0f5eb9ffef9a671465a144df12943bf6c4164d979907217d195101583dbfbf83\": rpc error: code = NotFound desc = could not find container \"0f5eb9ffef9a671465a144df12943bf6c4164d979907217d195101583dbfbf83\": container with ID starting with 0f5eb9ffef9a671465a144df12943bf6c4164d979907217d195101583dbfbf83 not found: ID does not exist" Nov 02 02:02:35.574373 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:35.574338 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v"] Nov 02 02:02:35.834044 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:35.834000 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:35.838400 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:35.838373 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t"] Nov 02 02:02:35.843611 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:35.843585 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b71ba5-6c9845bdff-d4p2t"] Nov 02 02:02:35.988372 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:35.988337 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:36.538021 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:36.537986 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:36.679939 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:36.679903 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:37.453767 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:37.453734 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b201831-e055-4763-9400-bb68d825dc3e" path="/var/lib/kubelet/pods/6b201831-e055-4763-9400-bb68d825dc3e/volumes" Nov 02 02:02:37.983423 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:37.983328 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc"] Nov 02 02:02:38.094904 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:38.094527 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj"] Nov 02 02:02:38.343868 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:38.343836 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:38.485722 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:38.485693 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:38.943616 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:38.943585 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:39.083559 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:39.083530 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:39.409957 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:39.409924 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:39.663233 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:39.663166 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:40.350530 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:40.350484 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:40.465730 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:40.465689 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x" podUID="6909a85d-37c8-4f16-907d-a893731003c3" containerName="payload-processing" containerID="cri-o://4447259a88a1835fb993e4f6c2fded9305232ccde19398c7916d309661450d95" gracePeriod=25 Nov 02 02:02:40.504309 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:40.504284 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:40.575121 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:40.575042 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v" podUID="56409aed-4dd3-4b2a-962c-7ab8bf857fed" containerName="payload-pre-processing" containerID="cri-o://1f9c6cf0b677f92f83d3977bcfdc3cce8cae0b2b043f500dd8e162fcc6df01ad" gracePeriod=25 Nov 02 02:02:40.738294 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:40.738269 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x" Nov 02 02:02:40.817233 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:40.817203 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6909a85d-37c8-4f16-907d-a893731003c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6909a85d-37c8-4f16-907d-a893731003c3-plugins-config-volume\") pod \"6909a85d-37c8-4f16-907d-a893731003c3\" (UID: \"6909a85d-37c8-4f16-907d-a893731003c3\") " Nov 02 02:02:40.817396 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:40.817259 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6909a85d-37c8-4f16-907d-a893731003c3-kube-api-access-ngqwh\" (UniqueName: \"kubernetes.io/projected/6909a85d-37c8-4f16-907d-a893731003c3-kube-api-access-ngqwh\") pod \"6909a85d-37c8-4f16-907d-a893731003c3\" (UID: \"6909a85d-37c8-4f16-907d-a893731003c3\") " Nov 02 02:02:40.817570 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:40.817548 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6909a85d-37c8-4f16-907d-a893731003c3-plugins-config-volume" pod "6909a85d-37c8-4f16-907d-a893731003c3" (UID: "6909a85d-37c8-4f16-907d-a893731003c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:02:40.817865 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:40.817831 5044 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6909a85d-37c8-4f16-907d-a893731003c3-plugins-config-volume\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:40.819919 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:40.819878 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6909a85d-37c8-4f16-907d-a893731003c3-kube-api-access-ngqwh" pod "6909a85d-37c8-4f16-907d-a893731003c3" (UID: "6909a85d-37c8-4f16-907d-a893731003c3"). InnerVolumeSpecName "kube-api-access-ngqwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:02:40.842708 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:40.842685 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v" Nov 02 02:02:40.863474 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:40.863446 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:40.918510 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:40.918440 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56409aed-4dd3-4b2a-962c-7ab8bf857fed-kube-api-access-pdfg9\" (UniqueName: \"kubernetes.io/projected/56409aed-4dd3-4b2a-962c-7ab8bf857fed-kube-api-access-pdfg9\") pod \"56409aed-4dd3-4b2a-962c-7ab8bf857fed\" (UID: \"56409aed-4dd3-4b2a-962c-7ab8bf857fed\") " Nov 02 02:02:40.918728 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:40.918706 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56409aed-4dd3-4b2a-962c-7ab8bf857fed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56409aed-4dd3-4b2a-962c-7ab8bf857fed-plugins-config-volume\") pod \"56409aed-4dd3-4b2a-962c-7ab8bf857fed\" (UID: \"56409aed-4dd3-4b2a-962c-7ab8bf857fed\") " Nov 02 02:02:40.919035 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:40.919011 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56409aed-4dd3-4b2a-962c-7ab8bf857fed-plugins-config-volume" pod "56409aed-4dd3-4b2a-962c-7ab8bf857fed" (UID: "56409aed-4dd3-4b2a-962c-7ab8bf857fed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:02:40.919203 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:40.919189 5044 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56409aed-4dd3-4b2a-962c-7ab8bf857fed-plugins-config-volume\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:40.919249 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:40.919206 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngqwh\" (UniqueName: \"kubernetes.io/projected/6909a85d-37c8-4f16-907d-a893731003c3-kube-api-access-ngqwh\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:40.920548 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:40.920524 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56409aed-4dd3-4b2a-962c-7ab8bf857fed-kube-api-access-pdfg9" pod "56409aed-4dd3-4b2a-962c-7ab8bf857fed" (UID: "56409aed-4dd3-4b2a-962c-7ab8bf857fed"). InnerVolumeSpecName "kube-api-access-pdfg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:02:41.020312 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.020281 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdfg9\" (UniqueName: \"kubernetes.io/projected/56409aed-4dd3-4b2a-962c-7ab8bf857fed-kube-api-access-pdfg9\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:41.221514 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.221436 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp"] Nov 02 02:02:41.221695 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.221624 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" podUID="46e40d47-ab73-4064-a9a2-f399cbd38c88" containerName="maas-api" containerID="cri-o://f0eee60ce90122dfd97d94a3a495aad583cbefdab8586de5a2d16a613f5393ca" gracePeriod=30 Nov 02 02:02:41.322433 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.322397 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:41.529723 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.529691 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" Nov 02 02:02:41.547294 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.547265 5044 generic.go:356] "Generic (PLEG): container finished" podID="6909a85d-37c8-4f16-907d-a893731003c3" containerID="4447259a88a1835fb993e4f6c2fded9305232ccde19398c7916d309661450d95" exitCode=0 Nov 02 02:02:41.547437 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.547309 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x" Nov 02 02:02:41.547437 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.547348 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x" event={"ID":"6909a85d-37c8-4f16-907d-a893731003c3","Type":"ContainerDied","Data":"4447259a88a1835fb993e4f6c2fded9305232ccde19398c7916d309661450d95"} Nov 02 02:02:41.547437 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.547400 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x" event={"ID":"6909a85d-37c8-4f16-907d-a893731003c3","Type":"ContainerDied","Data":"5ea27d137bdc4e46d37812980b61e0b65817a73849a8e9bd4ef82d0dd8b95309"} Nov 02 02:02:41.547437 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.547423 5044 scope.go:122] "RemoveContainer" containerID="4447259a88a1835fb993e4f6c2fded9305232ccde19398c7916d309661450d95" Nov 02 02:02:41.549075 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.549052 5044 generic.go:356] "Generic (PLEG): container finished" podID="56409aed-4dd3-4b2a-962c-7ab8bf857fed" containerID="1f9c6cf0b677f92f83d3977bcfdc3cce8cae0b2b043f500dd8e162fcc6df01ad" exitCode=0 Nov 02 02:02:41.549200 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.549095 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v" Nov 02 02:02:41.549200 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.549118 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v" event={"ID":"56409aed-4dd3-4b2a-962c-7ab8bf857fed","Type":"ContainerDied","Data":"1f9c6cf0b677f92f83d3977bcfdc3cce8cae0b2b043f500dd8e162fcc6df01ad"} Nov 02 02:02:41.549200 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.549164 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v" event={"ID":"56409aed-4dd3-4b2a-962c-7ab8bf857fed","Type":"ContainerDied","Data":"7d45698d4a10c47d2ef19f62dbbf2448c6202465fa31f7b60a5fa7a5fa0b3366"} Nov 02 02:02:41.551069 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.551047 5044 generic.go:356] "Generic (PLEG): container finished" podID="46e40d47-ab73-4064-a9a2-f399cbd38c88" containerID="f0eee60ce90122dfd97d94a3a495aad583cbefdab8586de5a2d16a613f5393ca" exitCode=0 Nov 02 02:02:41.551231 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.551102 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" Nov 02 02:02:41.551231 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.551120 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" event={"ID":"46e40d47-ab73-4064-a9a2-f399cbd38c88","Type":"ContainerDied","Data":"f0eee60ce90122dfd97d94a3a495aad583cbefdab8586de5a2d16a613f5393ca"} Nov 02 02:02:41.551231 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.551161 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp" event={"ID":"46e40d47-ab73-4064-a9a2-f399cbd38c88","Type":"ContainerDied","Data":"672ab0c8ad411460b51e555ee7744a5e6c8d4353aae4745b1930a7613dfe1cda"} Nov 02 02:02:41.558559 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.558541 5044 scope.go:122] "RemoveContainer" containerID="4447259a88a1835fb993e4f6c2fded9305232ccde19398c7916d309661450d95" Nov 02 02:02:41.558887 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:02:41.558867 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4447259a88a1835fb993e4f6c2fded9305232ccde19398c7916d309661450d95\": container with ID starting with 4447259a88a1835fb993e4f6c2fded9305232ccde19398c7916d309661450d95 not found: ID does not exist" containerID="4447259a88a1835fb993e4f6c2fded9305232ccde19398c7916d309661450d95" Nov 02 02:02:41.558979 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.558897 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4447259a88a1835fb993e4f6c2fded9305232ccde19398c7916d309661450d95"} err="failed to get container status \"4447259a88a1835fb993e4f6c2fded9305232ccde19398c7916d309661450d95\": rpc error: code = NotFound desc = could not find container \"4447259a88a1835fb993e4f6c2fded9305232ccde19398c7916d309661450d95\": container with ID starting with 4447259a88a1835fb993e4f6c2fded9305232ccde19398c7916d309661450d95 not found: ID does not exist" Nov 02 02:02:41.558979 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.558920 5044 scope.go:122] "RemoveContainer" containerID="1f9c6cf0b677f92f83d3977bcfdc3cce8cae0b2b043f500dd8e162fcc6df01ad" Nov 02 02:02:41.567791 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.567775 5044 scope.go:122] "RemoveContainer" containerID="1f9c6cf0b677f92f83d3977bcfdc3cce8cae0b2b043f500dd8e162fcc6df01ad" Nov 02 02:02:41.568053 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:02:41.568034 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f9c6cf0b677f92f83d3977bcfdc3cce8cae0b2b043f500dd8e162fcc6df01ad\": container with ID starting with 1f9c6cf0b677f92f83d3977bcfdc3cce8cae0b2b043f500dd8e162fcc6df01ad not found: ID does not exist" containerID="1f9c6cf0b677f92f83d3977bcfdc3cce8cae0b2b043f500dd8e162fcc6df01ad" Nov 02 02:02:41.568104 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.568060 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9c6cf0b677f92f83d3977bcfdc3cce8cae0b2b043f500dd8e162fcc6df01ad"} err="failed to get container status \"1f9c6cf0b677f92f83d3977bcfdc3cce8cae0b2b043f500dd8e162fcc6df01ad\": rpc error: code = NotFound desc = could not find container \"1f9c6cf0b677f92f83d3977bcfdc3cce8cae0b2b043f500dd8e162fcc6df01ad\": container with ID starting with 1f9c6cf0b677f92f83d3977bcfdc3cce8cae0b2b043f500dd8e162fcc6df01ad not found: ID does not exist" Nov 02 02:02:41.568104 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.568076 5044 scope.go:122] "RemoveContainer" containerID="f0eee60ce90122dfd97d94a3a495aad583cbefdab8586de5a2d16a613f5393ca" Nov 02 02:02:41.576745 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.576723 5044 scope.go:122] "RemoveContainer" containerID="f0eee60ce90122dfd97d94a3a495aad583cbefdab8586de5a2d16a613f5393ca" Nov 02 02:02:41.577015 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:02:41.576997 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0eee60ce90122dfd97d94a3a495aad583cbefdab8586de5a2d16a613f5393ca\": container with ID starting with f0eee60ce90122dfd97d94a3a495aad583cbefdab8586de5a2d16a613f5393ca not found: ID does not exist" containerID="f0eee60ce90122dfd97d94a3a495aad583cbefdab8586de5a2d16a613f5393ca" Nov 02 02:02:41.577070 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.577023 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0eee60ce90122dfd97d94a3a495aad583cbefdab8586de5a2d16a613f5393ca"} err="failed to get container status \"f0eee60ce90122dfd97d94a3a495aad583cbefdab8586de5a2d16a613f5393ca\": rpc error: code = NotFound desc = could not find container \"f0eee60ce90122dfd97d94a3a495aad583cbefdab8586de5a2d16a613f5393ca\": container with ID starting with f0eee60ce90122dfd97d94a3a495aad583cbefdab8586de5a2d16a613f5393ca not found: ID does not exist" Nov 02 02:02:41.626440 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.626418 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e40d47-ab73-4064-a9a2-f399cbd38c88-kube-api-access-2ndxr\" (UniqueName: \"kubernetes.io/projected/46e40d47-ab73-4064-a9a2-f399cbd38c88-kube-api-access-2ndxr\") pod \"46e40d47-ab73-4064-a9a2-f399cbd38c88\" (UID: \"46e40d47-ab73-4064-a9a2-f399cbd38c88\") " Nov 02 02:02:41.626549 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.626532 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46e40d47-ab73-4064-a9a2-f399cbd38c88-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46e40d47-ab73-4064-a9a2-f399cbd38c88-openshift-service-ca\") pod \"46e40d47-ab73-4064-a9a2-f399cbd38c88\" (UID: \"46e40d47-ab73-4064-a9a2-f399cbd38c88\") " Nov 02 02:02:41.626604 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.626587 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46e40d47-ab73-4064-a9a2-f399cbd38c88-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46e40d47-ab73-4064-a9a2-f399cbd38c88-maas-api-tls\") pod \"46e40d47-ab73-4064-a9a2-f399cbd38c88\" (UID: \"46e40d47-ab73-4064-a9a2-f399cbd38c88\") " Nov 02 02:02:41.626818 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.626800 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46e40d47-ab73-4064-a9a2-f399cbd38c88-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e40d47-ab73-4064-a9a2-f399cbd38c88-metrics-certs\") pod \"46e40d47-ab73-4064-a9a2-f399cbd38c88\" (UID: \"46e40d47-ab73-4064-a9a2-f399cbd38c88\") " Nov 02 02:02:41.626933 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.626906 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46e40d47-ab73-4064-a9a2-f399cbd38c88-openshift-service-ca" pod "46e40d47-ab73-4064-a9a2-f399cbd38c88" (UID: "46e40d47-ab73-4064-a9a2-f399cbd38c88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:02:41.627256 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.627237 5044 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46e40d47-ab73-4064-a9a2-f399cbd38c88-openshift-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:41.628529 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.628503 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e40d47-ab73-4064-a9a2-f399cbd38c88-kube-api-access-2ndxr" pod "46e40d47-ab73-4064-a9a2-f399cbd38c88" (UID: "46e40d47-ab73-4064-a9a2-f399cbd38c88"). InnerVolumeSpecName "kube-api-access-2ndxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:02:41.628775 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.628746 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46e40d47-ab73-4064-a9a2-f399cbd38c88-maas-api-tls" pod "46e40d47-ab73-4064-a9a2-f399cbd38c88" (UID: "46e40d47-ab73-4064-a9a2-f399cbd38c88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:02:41.628775 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.628764 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46e40d47-ab73-4064-a9a2-f399cbd38c88-metrics-certs" pod "46e40d47-ab73-4064-a9a2-f399cbd38c88" (UID: "46e40d47-ab73-4064-a9a2-f399cbd38c88"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:02:41.728188 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.728160 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46e40d47-ab73-4064-a9a2-f399cbd38c88-maas-api-tls\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:41.728188 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.728190 5044 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e40d47-ab73-4064-a9a2-f399cbd38c88-metrics-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:41.728391 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.728205 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ndxr\" (UniqueName: \"kubernetes.io/projected/46e40d47-ab73-4064-a9a2-f399cbd38c88-kube-api-access-2ndxr\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:41.767229 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.767200 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x"] Nov 02 02:02:41.774198 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.774168 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69c15d-58ffbcb58f-5xf6x"] Nov 02 02:02:41.861448 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.861403 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:41.940133 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.940101 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v"] Nov 02 02:02:41.946992 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:41.946965 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-69c15d-6648954d55-dn67v"] Nov 02 02:02:42.002711 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:42.002684 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:42.055813 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:42.055750 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp"] Nov 02 02:02:42.059861 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:42.059839 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-bb0464-5484485dcf-ds7hp"] Nov 02 02:02:42.983902 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:42.983864 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc" podUID="2135d51a-e3c9-43fe-956b-f49e6a5b3efe" containerName="payload-processing" containerID="cri-o://01f81ebe3238b5a2aadb835b617da775746ba200df7eb63714b6c28c996e1332" gracePeriod=25 Nov 02 02:02:43.095745 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.095666 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj" podUID="4c5bcecb-ef1c-4233-98b4-6461ff9ee181" containerName="payload-pre-processing" containerID="cri-o://4ec6120e463c2c53ea1620fca29916adab30eb5250cafd1542ba7587f985e4ff" gracePeriod=25 Nov 02 02:02:43.285568 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.285187 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc" Nov 02 02:02:43.396273 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.396239 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:43.399507 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.399482 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj" Nov 02 02:02:43.452804 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.452772 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46e40d47-ab73-4064-a9a2-f399cbd38c88" path="/var/lib/kubelet/pods/46e40d47-ab73-4064-a9a2-f399cbd38c88/volumes" Nov 02 02:02:43.453412 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.453394 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56409aed-4dd3-4b2a-962c-7ab8bf857fed" path="/var/lib/kubelet/pods/56409aed-4dd3-4b2a-962c-7ab8bf857fed/volumes" Nov 02 02:02:43.453973 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.453953 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6909a85d-37c8-4f16-907d-a893731003c3" path="/var/lib/kubelet/pods/6909a85d-37c8-4f16-907d-a893731003c3/volumes" Nov 02 02:02:43.455963 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.455941 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2135d51a-e3c9-43fe-956b-f49e6a5b3efe-kube-api-access-pj7fq\" (UniqueName: \"kubernetes.io/projected/2135d51a-e3c9-43fe-956b-f49e6a5b3efe-kube-api-access-pj7fq\") pod \"2135d51a-e3c9-43fe-956b-f49e6a5b3efe\" (UID: \"2135d51a-e3c9-43fe-956b-f49e6a5b3efe\") " Nov 02 02:02:43.456100 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.456083 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2135d51a-e3c9-43fe-956b-f49e6a5b3efe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2135d51a-e3c9-43fe-956b-f49e6a5b3efe-plugins-config-volume\") pod \"2135d51a-e3c9-43fe-956b-f49e6a5b3efe\" (UID: \"2135d51a-e3c9-43fe-956b-f49e6a5b3efe\") " Nov 02 02:02:43.457909 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.457883 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2135d51a-e3c9-43fe-956b-f49e6a5b3efe-plugins-config-volume" pod "2135d51a-e3c9-43fe-956b-f49e6a5b3efe" (UID: "2135d51a-e3c9-43fe-956b-f49e6a5b3efe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:02:43.458507 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.458477 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2135d51a-e3c9-43fe-956b-f49e6a5b3efe-kube-api-access-pj7fq" pod "2135d51a-e3c9-43fe-956b-f49e6a5b3efe" (UID: "2135d51a-e3c9-43fe-956b-f49e6a5b3efe"). InnerVolumeSpecName "kube-api-access-pj7fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:02:43.557438 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.557405 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c5bcecb-ef1c-4233-98b4-6461ff9ee181-kube-api-access-2bwqw\" (UniqueName: \"kubernetes.io/projected/4c5bcecb-ef1c-4233-98b4-6461ff9ee181-kube-api-access-2bwqw\") pod \"4c5bcecb-ef1c-4233-98b4-6461ff9ee181\" (UID: \"4c5bcecb-ef1c-4233-98b4-6461ff9ee181\") " Nov 02 02:02:43.557629 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.557464 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c5bcecb-ef1c-4233-98b4-6461ff9ee181-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c5bcecb-ef1c-4233-98b4-6461ff9ee181-plugins-config-volume\") pod \"4c5bcecb-ef1c-4233-98b4-6461ff9ee181\" (UID: \"4c5bcecb-ef1c-4233-98b4-6461ff9ee181\") " Nov 02 02:02:43.558179 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.557761 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c5bcecb-ef1c-4233-98b4-6461ff9ee181-plugins-config-volume" pod "4c5bcecb-ef1c-4233-98b4-6461ff9ee181" (UID: "4c5bcecb-ef1c-4233-98b4-6461ff9ee181"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:02:43.558291 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.558190 5044 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c5bcecb-ef1c-4233-98b4-6461ff9ee181-plugins-config-volume\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:43.558291 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.558214 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj7fq\" (UniqueName: \"kubernetes.io/projected/2135d51a-e3c9-43fe-956b-f49e6a5b3efe-kube-api-access-pj7fq\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:43.558291 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.558237 5044 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2135d51a-e3c9-43fe-956b-f49e6a5b3efe-plugins-config-volume\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:43.559752 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.559729 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5bcecb-ef1c-4233-98b4-6461ff9ee181-kube-api-access-2bwqw" pod "4c5bcecb-ef1c-4233-98b4-6461ff9ee181" (UID: "4c5bcecb-ef1c-4233-98b4-6461ff9ee181"). InnerVolumeSpecName "kube-api-access-2bwqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:02:43.569183 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.569156 5044 generic.go:356] "Generic (PLEG): container finished" podID="4c5bcecb-ef1c-4233-98b4-6461ff9ee181" containerID="4ec6120e463c2c53ea1620fca29916adab30eb5250cafd1542ba7587f985e4ff" exitCode=0 Nov 02 02:02:43.569280 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.569202 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj" Nov 02 02:02:43.569353 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.569287 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj" event={"ID":"4c5bcecb-ef1c-4233-98b4-6461ff9ee181","Type":"ContainerDied","Data":"4ec6120e463c2c53ea1620fca29916adab30eb5250cafd1542ba7587f985e4ff"} Nov 02 02:02:43.569414 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.569358 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj" event={"ID":"4c5bcecb-ef1c-4233-98b4-6461ff9ee181","Type":"ContainerDied","Data":"102d2f38441c379fe560a6a65a1dd6452808fc1734290a4b9003b517af750791"} Nov 02 02:02:43.569414 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.569392 5044 scope.go:122] "RemoveContainer" containerID="4ec6120e463c2c53ea1620fca29916adab30eb5250cafd1542ba7587f985e4ff" Nov 02 02:02:43.571041 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.571012 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc" event={"ID":"2135d51a-e3c9-43fe-956b-f49e6a5b3efe","Type":"ContainerDied","Data":"01f81ebe3238b5a2aadb835b617da775746ba200df7eb63714b6c28c996e1332"} Nov 02 02:02:43.571041 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.571038 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc" Nov 02 02:02:43.571172 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.570968 5044 generic.go:356] "Generic (PLEG): container finished" podID="2135d51a-e3c9-43fe-956b-f49e6a5b3efe" containerID="01f81ebe3238b5a2aadb835b617da775746ba200df7eb63714b6c28c996e1332" exitCode=0 Nov 02 02:02:43.571209 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.571177 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc" event={"ID":"2135d51a-e3c9-43fe-956b-f49e6a5b3efe","Type":"ContainerDied","Data":"d3287d78d349751b566de54e071c83ea064de9ec7596740a9502ffc4efc57f22"} Nov 02 02:02:43.582149 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.582126 5044 scope.go:122] "RemoveContainer" containerID="4ec6120e463c2c53ea1620fca29916adab30eb5250cafd1542ba7587f985e4ff" Nov 02 02:02:43.582465 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:02:43.582445 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec6120e463c2c53ea1620fca29916adab30eb5250cafd1542ba7587f985e4ff\": container with ID starting with 4ec6120e463c2c53ea1620fca29916adab30eb5250cafd1542ba7587f985e4ff not found: ID does not exist" containerID="4ec6120e463c2c53ea1620fca29916adab30eb5250cafd1542ba7587f985e4ff" Nov 02 02:02:43.582519 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.582472 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec6120e463c2c53ea1620fca29916adab30eb5250cafd1542ba7587f985e4ff"} err="failed to get container status \"4ec6120e463c2c53ea1620fca29916adab30eb5250cafd1542ba7587f985e4ff\": rpc error: code = NotFound desc = could not find container \"4ec6120e463c2c53ea1620fca29916adab30eb5250cafd1542ba7587f985e4ff\": container with ID starting with 4ec6120e463c2c53ea1620fca29916adab30eb5250cafd1542ba7587f985e4ff not found: ID does not exist" Nov 02 02:02:43.582519 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.582491 5044 scope.go:122] "RemoveContainer" containerID="01f81ebe3238b5a2aadb835b617da775746ba200df7eb63714b6c28c996e1332" Nov 02 02:02:43.591808 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.591790 5044 scope.go:122] "RemoveContainer" containerID="01f81ebe3238b5a2aadb835b617da775746ba200df7eb63714b6c28c996e1332" Nov 02 02:02:43.592076 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:02:43.592057 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f81ebe3238b5a2aadb835b617da775746ba200df7eb63714b6c28c996e1332\": container with ID starting with 01f81ebe3238b5a2aadb835b617da775746ba200df7eb63714b6c28c996e1332 not found: ID does not exist" containerID="01f81ebe3238b5a2aadb835b617da775746ba200df7eb63714b6c28c996e1332" Nov 02 02:02:43.592153 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.592082 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f81ebe3238b5a2aadb835b617da775746ba200df7eb63714b6c28c996e1332"} err="failed to get container status \"01f81ebe3238b5a2aadb835b617da775746ba200df7eb63714b6c28c996e1332\": rpc error: code = NotFound desc = could not find container \"01f81ebe3238b5a2aadb835b617da775746ba200df7eb63714b6c28c996e1332\": container with ID starting with 01f81ebe3238b5a2aadb835b617da775746ba200df7eb63714b6c28c996e1332 not found: ID does not exist" Nov 02 02:02:43.659014 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.658985 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bwqw\" (UniqueName: \"kubernetes.io/projected/4c5bcecb-ef1c-4233-98b4-6461ff9ee181-kube-api-access-2bwqw\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:02:43.701148 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.701121 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj"] Nov 02 02:02:43.705365 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.705345 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-269235-5696dc46f9-bnpwj"] Nov 02 02:02:43.810919 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.810783 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc"] Nov 02 02:02:43.814308 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:43.814280 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-269235-7cb45b6d5c-wr5fc"] Nov 02 02:02:45.452551 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:45.452517 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2135d51a-e3c9-43fe-956b-f49e6a5b3efe" path="/var/lib/kubelet/pods/2135d51a-e3c9-43fe-956b-f49e6a5b3efe/volumes" Nov 02 02:02:45.453000 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:45.452984 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c5bcecb-ef1c-4233-98b4-6461ff9ee181" path="/var/lib/kubelet/pods/4c5bcecb-ef1c-4233-98b4-6461ff9ee181/volumes" Nov 02 02:02:45.940926 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:45.940894 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:46.133954 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:46.133910 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:46.276198 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:46.276129 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:46.910949 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:46.910869 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:47.118959 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:47.118923 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:47.268441 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:47.268412 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:47.805289 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:47.805249 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:49.528193 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:49.528164 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:49.669939 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:49.669907 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:49.809934 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:49.809854 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:50.462114 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:50.462076 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:52.230856 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:52.230825 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:52.376431 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:52.376399 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:53.598786 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:53.598706 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:53.745413 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:53.745362 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:53.936899 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:53.936816 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:54.173022 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:02:54.172984 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:02:54.429141 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:54.429106 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:54.854556 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:54.854526 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:56.355556 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:56.355524 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:56.569873 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:56.569835 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:56.773810 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:56.773781 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:57.636143 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:57.636111 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:58.279879 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:58.279841 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:58.602392 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:58.602281 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:59.396362 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.396322 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b"] Nov 02 02:02:59.453180 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.453152 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22"] Nov 02 02:02:59.512295 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.512269 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:02:59.997562 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.997531 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5"] Nov 02 02:02:59.998094 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998077 5044 cpu_manager.go:414] "containerMap: removing container" podUID="54c547c6-30fd-43be-ac27-299da9065a25" containerName="istio-proxy" Nov 02 02:02:59.998094 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998095 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c547c6-30fd-43be-ac27-299da9065a25" containerName="istio-proxy" Nov 02 02:02:59.998210 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998111 5044 cpu_manager.go:414] "containerMap: removing container" podUID="46e40d47-ab73-4064-a9a2-f399cbd38c88" containerName="maas-api" Nov 02 02:02:59.998210 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998117 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e40d47-ab73-4064-a9a2-f399cbd38c88" containerName="maas-api" Nov 02 02:02:59.998210 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998125 5044 cpu_manager.go:414] "containerMap: removing container" podUID="6909a85d-37c8-4f16-907d-a893731003c3" containerName="payload-processing" Nov 02 02:02:59.998210 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998131 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6909a85d-37c8-4f16-907d-a893731003c3" containerName="payload-processing" Nov 02 02:02:59.998210 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998140 5044 cpu_manager.go:414] "containerMap: removing container" podUID="56409aed-4dd3-4b2a-962c-7ab8bf857fed" containerName="payload-pre-processing" Nov 02 02:02:59.998210 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998145 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56409aed-4dd3-4b2a-962c-7ab8bf857fed" containerName="payload-pre-processing" Nov 02 02:02:59.998210 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998155 5044 cpu_manager.go:414] "containerMap: removing container" podUID="2135d51a-e3c9-43fe-956b-f49e6a5b3efe" containerName="payload-processing" Nov 02 02:02:59.998210 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998160 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2135d51a-e3c9-43fe-956b-f49e6a5b3efe" containerName="payload-processing" Nov 02 02:02:59.998210 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998174 5044 cpu_manager.go:414] "containerMap: removing container" podUID="e0072c75-d48f-4a49-a1ba-5586e673fd36" containerName="maas-api" Nov 02 02:02:59.998210 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998178 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0072c75-d48f-4a49-a1ba-5586e673fd36" containerName="maas-api" Nov 02 02:02:59.998210 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998184 5044 cpu_manager.go:414] "containerMap: removing container" podUID="6b201831-e055-4763-9400-bb68d825dc3e" containerName="payload-pre-processing" Nov 02 02:02:59.998210 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998189 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b201831-e055-4763-9400-bb68d825dc3e" containerName="payload-pre-processing" Nov 02 02:02:59.998210 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998200 5044 cpu_manager.go:414] "containerMap: removing container" podUID="fcdbd493-2074-4976-aef4-9eb91f6f7522" containerName="payload-processing" Nov 02 02:02:59.998210 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998205 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcdbd493-2074-4976-aef4-9eb91f6f7522" containerName="payload-processing" Nov 02 02:02:59.998210 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998213 5044 cpu_manager.go:414] "containerMap: removing container" podUID="66b4726e-ea0f-46e5-b1a8-cf803f344ef4" containerName="maas-api" Nov 02 02:02:59.998210 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998218 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b4726e-ea0f-46e5-b1a8-cf803f344ef4" containerName="maas-api" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998235 5044 cpu_manager.go:414] "containerMap: removing container" podUID="4c5bcecb-ef1c-4233-98b4-6461ff9ee181" containerName="payload-pre-processing" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998241 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c5bcecb-ef1c-4233-98b4-6461ff9ee181" containerName="payload-pre-processing" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998250 5044 cpu_manager.go:414] "containerMap: removing container" podUID="e68f185a-890d-445d-9e1e-b35d7d0e2448" containerName="maas-api" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998255 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68f185a-890d-445d-9e1e-b35d7d0e2448" containerName="maas-api" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998262 5044 cpu_manager.go:414] "containerMap: removing container" podUID="8ee58b67-0d93-4541-b43d-0c8f790ecc38" containerName="payload-pre-processing" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998267 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee58b67-0d93-4541-b43d-0c8f790ecc38" containerName="payload-pre-processing" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998279 5044 cpu_manager.go:414] "containerMap: removing container" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998283 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998291 5044 cpu_manager.go:414] "containerMap: removing container" podUID="a83df443-031e-4a2e-a4f4-5a686ad12085" containerName="maas-api" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998295 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83df443-031e-4a2e-a4f4-5a686ad12085" containerName="maas-api" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998305 5044 cpu_manager.go:414] "containerMap: removing container" podUID="1098cb6c-8b36-4df8-87d9-357d3ed510e0" containerName="istio-proxy" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998310 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1098cb6c-8b36-4df8-87d9-357d3ed510e0" containerName="istio-proxy" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998319 5044 cpu_manager.go:414] "containerMap: removing container" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998324 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998330 5044 cpu_manager.go:414] "containerMap: removing container" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998334 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998341 5044 cpu_manager.go:414] "containerMap: removing container" podUID="1098cb6c-8b36-4df8-87d9-357d3ed510e0" containerName="istio-proxy" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998345 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1098cb6c-8b36-4df8-87d9-357d3ed510e0" containerName="istio-proxy" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998434 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e6f149-8398-4eb5-8fcc-f658c6e3300c" containerName="main" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998443 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83df443-031e-4a2e-a4f4-5a686ad12085" containerName="maas-api" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998449 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="1098cb6c-8b36-4df8-87d9-357d3ed510e0" containerName="istio-proxy" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998455 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="cada31e5-25a0-464e-9e89-f857de57beba" containerName="main" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998461 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad28d87-78de-4b69-9f2a-4d713cbac5ad" containerName="main" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998465 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="1098cb6c-8b36-4df8-87d9-357d3ed510e0" containerName="istio-proxy" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998471 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c547c6-30fd-43be-ac27-299da9065a25" containerName="istio-proxy" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998477 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e40d47-ab73-4064-a9a2-f399cbd38c88" containerName="maas-api" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998483 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="6909a85d-37c8-4f16-907d-a893731003c3" containerName="payload-processing" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998489 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="56409aed-4dd3-4b2a-962c-7ab8bf857fed" containerName="payload-pre-processing" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998496 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="2135d51a-e3c9-43fe-956b-f49e6a5b3efe" containerName="payload-processing" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998504 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0072c75-d48f-4a49-a1ba-5586e673fd36" containerName="maas-api" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998510 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b201831-e055-4763-9400-bb68d825dc3e" containerName="payload-pre-processing" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998516 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcdbd493-2074-4976-aef4-9eb91f6f7522" containerName="payload-processing" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998522 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b4726e-ea0f-46e5-b1a8-cf803f344ef4" containerName="maas-api" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998527 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c5bcecb-ef1c-4233-98b4-6461ff9ee181" containerName="payload-pre-processing" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998532 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68f185a-890d-445d-9e1e-b35d7d0e2448" containerName="maas-api" Nov 02 02:02:59.998812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:02:59.998537 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee58b67-0d93-4541-b43d-0c8f790ecc38" containerName="payload-pre-processing" Nov 02 02:03:00.006484 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:00.006438 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" Nov 02 02:03:00.061574 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:00.061541 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5"] Nov 02 02:03:00.076157 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:00.076124 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-07ac70d0-serving-cert\"" Nov 02 02:03:00.098944 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:00.098913 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6txb\" (UniqueName: \"kubernetes.io/projected/f20c0468-c6b6-4718-b3f2-718c8c044fb9-kube-api-access-d6txb\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5\" (UID: \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" Nov 02 02:03:00.099113 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:00.099030 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f20c0468-c6b6-4718-b3f2-718c8c044fb9-openshift-service-ca\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5\" (UID: \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" Nov 02 02:03:00.099113 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:00.099052 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f20c0468-c6b6-4718-b3f2-718c8c044fb9-maas-api-tls\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5\" (UID: \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" Nov 02 02:03:00.099113 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:00.099086 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f20c0468-c6b6-4718-b3f2-718c8c044fb9-metrics-certs\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5\" (UID: \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" Nov 02 02:03:00.147658 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:00.147618 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:00.202960 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:00.202927 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f20c0468-c6b6-4718-b3f2-718c8c044fb9-openshift-service-ca\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5\" (UID: \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" Nov 02 02:03:00.203143 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:00.202974 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f20c0468-c6b6-4718-b3f2-718c8c044fb9-maas-api-tls\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5\" (UID: \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" Nov 02 02:03:00.203143 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:00.203032 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f20c0468-c6b6-4718-b3f2-718c8c044fb9-metrics-certs\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5\" (UID: \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" Nov 02 02:03:00.203143 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:00.203122 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6txb\" (UniqueName: \"kubernetes.io/projected/f20c0468-c6b6-4718-b3f2-718c8c044fb9-kube-api-access-d6txb\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5\" (UID: \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" Nov 02 02:03:00.204290 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:03:00.204267 5044 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-serving-cert: secret "maas-api-e2e-mt-07ac70d0-serving-cert" not found Nov 02 02:03:00.204496 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:03:00.204482 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20c0468-c6b6-4718-b3f2-718c8c044fb9-maas-api-tls podName:f20c0468-c6b6-4718-b3f2-718c8c044fb9 nodeName:}" failed. No retries permitted until 2026-11-02 02:03:00.704461656 +0000 UTC m=+2297.375714323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f20c0468-c6b6-4718-b3f2-718c8c044fb9-maas-api-tls") pod "maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" (UID: "f20c0468-c6b6-4718-b3f2-718c8c044fb9") : secret "maas-api-e2e-mt-07ac70d0-serving-cert" not found Nov 02 02:03:00.205582 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:00.205560 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f20c0468-c6b6-4718-b3f2-718c8c044fb9-openshift-service-ca\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5\" (UID: \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" Nov 02 02:03:00.225307 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:00.225269 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f20c0468-c6b6-4718-b3f2-718c8c044fb9-metrics-certs\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5\" (UID: \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" Nov 02 02:03:00.265346 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:00.265321 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6txb\" (UniqueName: \"kubernetes.io/projected/f20c0468-c6b6-4718-b3f2-718c8c044fb9-kube-api-access-d6txb\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5\" (UID: \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" Nov 02 02:03:00.361211 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:00.361172 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:00.504466 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:00.504434 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:00.709461 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:00.709381 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f20c0468-c6b6-4718-b3f2-718c8c044fb9-maas-api-tls\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5\" (UID: \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" Nov 02 02:03:00.712224 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:00.712199 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f20c0468-c6b6-4718-b3f2-718c8c044fb9-maas-api-tls\") pod \"maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5\" (UID: \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" Nov 02 02:03:00.946103 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:00.946075 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" Nov 02 02:03:01.325883 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:01.325851 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5"] Nov 02 02:03:01.326324 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:03:01.326297 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20c0468_c6b6_4718_b3f2_718c8c044fb9.slice/crio-a473341affca3c6e4f6bdd7c2f696bc3a21ddc7d35df33b6f21ef4d7de74f8ff WatchSource:0}: Error finding container a473341affca3c6e4f6bdd7c2f696bc3a21ddc7d35df33b6f21ef4d7de74f8ff: Status 404 returned error can't find the container with id a473341affca3c6e4f6bdd7c2f696bc3a21ddc7d35df33b6f21ef4d7de74f8ff Nov 02 02:03:01.328138 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:01.328116 5044 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 02:03:01.683547 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:01.683457 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" event={"ID":"f20c0468-c6b6-4718-b3f2-718c8c044fb9","Type":"ContainerStarted","Data":"a473341affca3c6e4f6bdd7c2f696bc3a21ddc7d35df33b6f21ef4d7de74f8ff"} Nov 02 02:03:02.691526 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:02.691486 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" event={"ID":"f20c0468-c6b6-4718-b3f2-718c8c044fb9","Type":"ContainerStarted","Data":"7bfd0b48706bd64dee09cf5f1a1f2f35e66495704b3401c38457f308c92b2356"} Nov 02 02:03:02.691994 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:02.691692 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" Nov 02 02:03:02.699060 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:02.699033 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:02.811216 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:02.811147 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" podStartSLOduration=2.940893112 podStartE2EDuration="3.811132491s" totalImagesPullingTime="870.239379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:03:02.81023778 +0000 UTC m=+2299.481490464" watchObservedRunningTime="2026-11-02 02:03:02.811132491 +0000 UTC m=+2299.482385172" Nov 02 02:03:02.867886 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:02.867854 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:03.025258 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:03.025218 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:05.586209 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:05.586177 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:06.097343 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:06.097315 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:06.269830 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:06.269800 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:07.691618 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:07.691575 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:07.889923 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:07.889873 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:08.705396 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:08.705368 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" Nov 02 02:03:10.982405 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:10.982372 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:14.419098 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.419047 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" containerID="cri-o://2a51530d5b80a2dc8a3768ce449c36e826b71766b01f37c849ec9258991bfc3c" gracePeriod=45 Nov 02 02:03:14.473433 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.473397 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" containerID="cri-o://c8d360b801611315f2ad338dd55c5dad45bdd99cf73c7fa0f556de4a3ddeb8a6" gracePeriod=45 Nov 02 02:03:14.761846 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.761820 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 02:03:14.773406 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.773371 5044 generic.go:356] "Generic (PLEG): container finished" podID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerID="c8d360b801611315f2ad338dd55c5dad45bdd99cf73c7fa0f556de4a3ddeb8a6" exitCode=0 Nov 02 02:03:14.773545 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.773473 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" event={"ID":"9a1c8fde-4ec7-433a-9f68-0470dee95c46","Type":"ContainerDied","Data":"c8d360b801611315f2ad338dd55c5dad45bdd99cf73c7fa0f556de4a3ddeb8a6"} Nov 02 02:03:14.775014 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.774991 5044 generic.go:356] "Generic (PLEG): container finished" podID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerID="2a51530d5b80a2dc8a3768ce449c36e826b71766b01f37c849ec9258991bfc3c" exitCode=0 Nov 02 02:03:14.775143 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.775038 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" Nov 02 02:03:14.775143 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.775075 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" event={"ID":"70609eac-cd4a-4e96-84b0-536dccf3a715","Type":"ContainerDied","Data":"2a51530d5b80a2dc8a3768ce449c36e826b71766b01f37c849ec9258991bfc3c"} Nov 02 02:03:14.775143 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.775116 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b" event={"ID":"70609eac-cd4a-4e96-84b0-536dccf3a715","Type":"ContainerDied","Data":"eac4152d5c8ef23fe776dfe66bbb6dc998d8af3b9a0ccf1c159a3674d7a3f53c"} Nov 02 02:03:14.775264 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.775150 5044 scope.go:122] "RemoveContainer" containerID="2a51530d5b80a2dc8a3768ce449c36e826b71766b01f37c849ec9258991bfc3c" Nov 02 02:03:14.786879 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.786858 5044 scope.go:122] "RemoveContainer" containerID="2a51530d5b80a2dc8a3768ce449c36e826b71766b01f37c849ec9258991bfc3c" Nov 02 02:03:14.787259 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:03:14.787102 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a51530d5b80a2dc8a3768ce449c36e826b71766b01f37c849ec9258991bfc3c\": container with ID starting with 2a51530d5b80a2dc8a3768ce449c36e826b71766b01f37c849ec9258991bfc3c not found: ID does not exist" containerID="2a51530d5b80a2dc8a3768ce449c36e826b71766b01f37c849ec9258991bfc3c" Nov 02 02:03:14.787259 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.787137 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a51530d5b80a2dc8a3768ce449c36e826b71766b01f37c849ec9258991bfc3c"} err="failed to get container status \"2a51530d5b80a2dc8a3768ce449c36e826b71766b01f37c849ec9258991bfc3c\": rpc error: code = NotFound desc = could not find container \"2a51530d5b80a2dc8a3768ce449c36e826b71766b01f37c849ec9258991bfc3c\": container with ID starting with 2a51530d5b80a2dc8a3768ce449c36e826b71766b01f37c849ec9258991bfc3c not found: ID does not exist" Nov 02 02:03:14.814294 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.814272 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 02:03:14.893889 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.892123 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-tmp-dir\") pod \"70609eac-cd4a-4e96-84b0-536dccf3a715\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " Nov 02 02:03:14.893889 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.892274 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70609eac-cd4a-4e96-84b0-536dccf3a715-tls-certs\" (UniqueName: \"kubernetes.io/secret/70609eac-cd4a-4e96-84b0-536dccf3a715-tls-certs\") pod \"70609eac-cd4a-4e96-84b0-536dccf3a715\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " Nov 02 02:03:14.893889 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.892958 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-tmp-dir" pod "70609eac-cd4a-4e96-84b0-536dccf3a715" (UID: "70609eac-cd4a-4e96-84b0-536dccf3a715"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:03:14.893889 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.893086 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-dshm\") pod \"70609eac-cd4a-4e96-84b0-536dccf3a715\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " Nov 02 02:03:14.893889 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.893160 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70609eac-cd4a-4e96-84b0-536dccf3a715-kube-api-access-2hpv4\" (UniqueName: \"kubernetes.io/projected/70609eac-cd4a-4e96-84b0-536dccf3a715-kube-api-access-2hpv4\") pod \"70609eac-cd4a-4e96-84b0-536dccf3a715\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " Nov 02 02:03:14.893889 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.893244 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-model-cache\") pod \"70609eac-cd4a-4e96-84b0-536dccf3a715\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " Nov 02 02:03:14.893889 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.893288 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-home\" (UniqueName: \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-home\") pod \"70609eac-cd4a-4e96-84b0-536dccf3a715\" (UID: \"70609eac-cd4a-4e96-84b0-536dccf3a715\") " Nov 02 02:03:14.893889 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.893783 5044 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-tmp-dir\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:03:14.894468 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.894290 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-home" pod "70609eac-cd4a-4e96-84b0-536dccf3a715" (UID: "70609eac-cd4a-4e96-84b0-536dccf3a715"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:03:14.894529 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.894508 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-model-cache" pod "70609eac-cd4a-4e96-84b0-536dccf3a715" (UID: "70609eac-cd4a-4e96-84b0-536dccf3a715"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:03:14.896942 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.896910 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-dshm" pod "70609eac-cd4a-4e96-84b0-536dccf3a715" (UID: "70609eac-cd4a-4e96-84b0-536dccf3a715"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:03:14.897192 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.897164 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70609eac-cd4a-4e96-84b0-536dccf3a715-kube-api-access-2hpv4" pod "70609eac-cd4a-4e96-84b0-536dccf3a715" (UID: "70609eac-cd4a-4e96-84b0-536dccf3a715"). InnerVolumeSpecName "kube-api-access-2hpv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:03:14.897339 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.897318 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70609eac-cd4a-4e96-84b0-536dccf3a715-tls-certs" pod "70609eac-cd4a-4e96-84b0-536dccf3a715" (UID: "70609eac-cd4a-4e96-84b0-536dccf3a715"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:03:14.995149 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.995058 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a1c8fde-4ec7-433a-9f68-0470dee95c46-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1c8fde-4ec7-433a-9f68-0470dee95c46-tls-certs\") pod \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " Nov 02 02:03:14.995149 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.995121 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-model-cache\") pod \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " Nov 02 02:03:14.995387 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.995201 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1c8fde-4ec7-433a-9f68-0470dee95c46-kube-api-access-s6kll\" (UniqueName: \"kubernetes.io/projected/9a1c8fde-4ec7-433a-9f68-0470dee95c46-kube-api-access-s6kll\") pod \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " Nov 02 02:03:14.995387 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.995305 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-home\" (UniqueName: \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-home\") pod \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " Nov 02 02:03:14.995503 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.995485 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-tmp-dir\") pod \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " Nov 02 02:03:14.995582 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.995515 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-home" pod "9a1c8fde-4ec7-433a-9f68-0470dee95c46" (UID: "9a1c8fde-4ec7-433a-9f68-0470dee95c46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:03:14.995582 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.995531 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-model-cache" pod "9a1c8fde-4ec7-433a-9f68-0470dee95c46" (UID: "9a1c8fde-4ec7-433a-9f68-0470dee95c46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:03:14.995582 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.995544 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-dshm\") pod \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\" (UID: \"9a1c8fde-4ec7-433a-9f68-0470dee95c46\") " Nov 02 02:03:14.995858 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.995702 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-tmp-dir" pod "9a1c8fde-4ec7-433a-9f68-0470dee95c46" (UID: "9a1c8fde-4ec7-433a-9f68-0470dee95c46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:03:14.996072 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.996045 5044 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70609eac-cd4a-4e96-84b0-536dccf3a715-tls-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:03:14.996177 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.996077 5044 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-model-cache\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:03:14.996177 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.996090 5044 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-dshm\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:03:14.996177 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.996104 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hpv4\" (UniqueName: \"kubernetes.io/projected/70609eac-cd4a-4e96-84b0-536dccf3a715-kube-api-access-2hpv4\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:03:14.996177 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.996119 5044 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-home\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:03:14.996177 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.996135 5044 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-model-cache\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:03:14.996177 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.996147 5044 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70609eac-cd4a-4e96-84b0-536dccf3a715-home\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:03:14.996177 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.996160 5044 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-tmp-dir\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:03:14.997581 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.997557 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-dshm" pod "9a1c8fde-4ec7-433a-9f68-0470dee95c46" (UID: "9a1c8fde-4ec7-433a-9f68-0470dee95c46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:03:14.997688 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.997584 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a1c8fde-4ec7-433a-9f68-0470dee95c46-tls-certs" pod "9a1c8fde-4ec7-433a-9f68-0470dee95c46" (UID: "9a1c8fde-4ec7-433a-9f68-0470dee95c46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:03:14.997871 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:14.997854 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1c8fde-4ec7-433a-9f68-0470dee95c46-kube-api-access-s6kll" pod "9a1c8fde-4ec7-433a-9f68-0470dee95c46" (UID: "9a1c8fde-4ec7-433a-9f68-0470dee95c46"). InnerVolumeSpecName "kube-api-access-s6kll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:03:15.098547 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:15.097637 5044 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1c8fde-4ec7-433a-9f68-0470dee95c46-dshm\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:03:15.098547 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:15.097702 5044 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1c8fde-4ec7-433a-9f68-0470dee95c46-tls-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:03:15.098547 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:15.097720 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6kll\" (UniqueName: \"kubernetes.io/projected/9a1c8fde-4ec7-433a-9f68-0470dee95c46-kube-api-access-s6kll\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:03:15.207296 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:15.207263 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b"] Nov 02 02:03:15.211430 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:15.211394 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-187478/distinct-w4-187478-kserve-76df86f488-mrd5b"] Nov 02 02:03:15.455175 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:15.455143 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" path="/var/lib/kubelet/pods/70609eac-cd4a-4e96-84b0-536dccf3a715/volumes" Nov 02 02:03:15.783598 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:15.783569 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" Nov 02 02:03:15.783598 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:15.783589 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22" event={"ID":"9a1c8fde-4ec7-433a-9f68-0470dee95c46","Type":"ContainerDied","Data":"9e296d4a87d85e0c2edf65b9958f192ad14ce4fbf9782d5d1624335f54ac65aa"} Nov 02 02:03:15.783885 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:15.783639 5044 scope.go:122] "RemoveContainer" containerID="c8d360b801611315f2ad338dd55c5dad45bdd99cf73c7fa0f556de4a3ddeb8a6" Nov 02 02:03:15.953200 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:15.953167 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22"] Nov 02 02:03:15.955865 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:15.955843 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-187478/premium-w4-187478-kserve-6496c99c69-fdv22"] Nov 02 02:03:16.170511 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:16.170420 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:17.453377 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:17.453345 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" path="/var/lib/kubelet/pods/9a1c8fde-4ec7-433a-9f68-0470dee95c46/volumes" Nov 02 02:03:20.919955 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:20.919925 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:24.172797 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:03:24.172762 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:03:25.318050 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:25.318017 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:26.845608 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:26.845562 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:27.069648 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:27.069601 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:27.220740 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:27.220620 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:27.740406 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:27.740372 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:31.514981 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:31.514939 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:32.683629 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:32.683591 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:33.057634 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:33.057591 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:33.450021 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:33.449947 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:34.008867 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:34.008830 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:34.150802 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:34.150775 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:34.297119 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:34.297040 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:34.488627 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:34.488592 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:34.649330 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:34.649225 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:35.340569 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:35.340538 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:36.497074 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:36.497038 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:36.685199 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:36.685156 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:37.154122 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:37.154039 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:37.473639 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:37.473545 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:38.376220 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:38.376185 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:39.925602 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:39.925561 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:40.400994 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:40.400914 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:40.548983 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:40.548946 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:40.692547 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:40.692470 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:41.167860 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:41.167827 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:41.337466 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:41.337424 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:41.489908 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:41.489819 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:42.158966 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:42.158929 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:42.307382 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:42.307347 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:42.454334 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:42.454236 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:42.624991 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:42.624959 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:43.265214 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:43.265181 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:43.892665 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:43.892623 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:44.156348 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:44.156233 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:44.747028 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:44.746980 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:44.890505 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:44.890477 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:45.259592 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:45.259560 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:46.457535 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:46.457472 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:47.510946 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:47.510915 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26"] Nov 02 02:03:47.561221 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:47.561192 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm"] Nov 02 02:03:47.637462 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:47.637432 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:48.208106 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:48.208073 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:48.411494 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:48.411460 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:48.560721 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:48.560635 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:50.474991 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:50.474957 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:54.176429 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:03:54.176390 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:03:55.552947 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:55.552919 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:55.753570 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:55.753541 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:55.896279 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:55.896204 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:57.856001 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:57.855959 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2"] Nov 02 02:03:58.211936 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:58.211863 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:03:58.354759 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:03:58.354729 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:01.016343 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:01.016259 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:02.533566 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:02.533526 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" containerID="cri-o://d5bc98f234a9b3c1d46aa172b8418d023df3fd1eadb6ff747a19890a00b06466" gracePeriod=45 Nov 02 02:04:02.581986 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:02.581914 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" containerID="cri-o://c3c8402ba52cfd2bc608c16c6c126655230f62d55aaf53cc366a57b3fec471c2" gracePeriod=45 Nov 02 02:04:02.856550 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:02.856503 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2" podUID="cfeb96d5-3e36-4855-b2e6-7cf5311d131b" containerName="payload-processing" containerID="cri-o://ec0fe4eab8143f04a4560c06c856130318149f1fa4cb028c9087632d10653123" gracePeriod=25 Nov 02 02:04:02.901210 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:02.901178 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 02:04:02.904286 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:02.904267 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 02:04:03.059174 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.059078 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e539f99-221e-4050-87f3-a7bd2d1fbb71-kube-api-access-d728t\" (UniqueName: \"kubernetes.io/projected/4e539f99-221e-4050-87f3-a7bd2d1fbb71-kube-api-access-d728t\") pod \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " Nov 02 02:04:03.059330 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.059169 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-tls-certs\") pod \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " Nov 02 02:04:03.059330 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.059248 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-model-cache\") pod \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " Nov 02 02:04:03.059330 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.059282 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-model-cache\") pod \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " Nov 02 02:04:03.059500 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.059341 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-tmp-dir\") pod \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " Nov 02 02:04:03.059500 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.059367 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-dshm\") pod \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " Nov 02 02:04:03.059500 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.059399 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e539f99-221e-4050-87f3-a7bd2d1fbb71-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e539f99-221e-4050-87f3-a7bd2d1fbb71-tls-certs\") pod \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " Nov 02 02:04:03.059500 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.059495 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-tmp-dir\") pod \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " Nov 02 02:04:03.059638 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.059566 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-home\" (UniqueName: \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-home\") pod \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " Nov 02 02:04:03.059638 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.059605 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-kube-api-access-8s2zd\" (UniqueName: \"kubernetes.io/projected/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-kube-api-access-8s2zd\") pod \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " Nov 02 02:04:03.059748 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.059686 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-home\" (UniqueName: \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-home\") pod \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\" (UID: \"4e539f99-221e-4050-87f3-a7bd2d1fbb71\") " Nov 02 02:04:03.059748 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.059713 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-dshm\") pod \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\" (UID: \"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94\") " Nov 02 02:04:03.060901 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.060805 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-home" pod "3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" (UID: "3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:04:03.061085 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.061016 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-home" pod "4e539f99-221e-4050-87f3-a7bd2d1fbb71" (UID: "4e539f99-221e-4050-87f3-a7bd2d1fbb71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:04:03.061560 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.061265 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-model-cache" pod "4e539f99-221e-4050-87f3-a7bd2d1fbb71" (UID: "4e539f99-221e-4050-87f3-a7bd2d1fbb71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:04:03.061560 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.061482 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-tmp-dir" pod "3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" (UID: "3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:04:03.061560 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.061541 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-model-cache" pod "3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" (UID: "3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:04:03.061846 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.061823 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-tmp-dir" pod "4e539f99-221e-4050-87f3-a7bd2d1fbb71" (UID: "4e539f99-221e-4050-87f3-a7bd2d1fbb71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:04:03.063454 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.063426 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-dshm" pod "4e539f99-221e-4050-87f3-a7bd2d1fbb71" (UID: "4e539f99-221e-4050-87f3-a7bd2d1fbb71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:04:03.064460 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.064433 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-kube-api-access-8s2zd" pod "3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" (UID: "3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94"). InnerVolumeSpecName "kube-api-access-8s2zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:04:03.065092 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.065051 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-dshm" pod "3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" (UID: "3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:04:03.065448 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.065422 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-tls-certs" pod "3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" (UID: "3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:04:03.065655 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.065636 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e539f99-221e-4050-87f3-a7bd2d1fbb71-tls-certs" pod "4e539f99-221e-4050-87f3-a7bd2d1fbb71" (UID: "4e539f99-221e-4050-87f3-a7bd2d1fbb71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:04:03.065770 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.065730 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e539f99-221e-4050-87f3-a7bd2d1fbb71-kube-api-access-d728t" pod "4e539f99-221e-4050-87f3-a7bd2d1fbb71" (UID: "4e539f99-221e-4050-87f3-a7bd2d1fbb71"). InnerVolumeSpecName "kube-api-access-d728t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:04:03.095100 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.095069 5044 generic.go:356] "Generic (PLEG): container finished" podID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerID="d5bc98f234a9b3c1d46aa172b8418d023df3fd1eadb6ff747a19890a00b06466" exitCode=0 Nov 02 02:04:03.095255 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.095115 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" Nov 02 02:04:03.095255 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.095152 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" event={"ID":"4e539f99-221e-4050-87f3-a7bd2d1fbb71","Type":"ContainerDied","Data":"d5bc98f234a9b3c1d46aa172b8418d023df3fd1eadb6ff747a19890a00b06466"} Nov 02 02:04:03.095255 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.095198 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26" event={"ID":"4e539f99-221e-4050-87f3-a7bd2d1fbb71","Type":"ContainerDied","Data":"08bb363191f2186441c941f17a0cfdf772b44820894120648cfda9e71e1065e7"} Nov 02 02:04:03.095255 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.095219 5044 scope.go:122] "RemoveContainer" containerID="d5bc98f234a9b3c1d46aa172b8418d023df3fd1eadb6ff747a19890a00b06466" Nov 02 02:04:03.096764 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.096735 5044 generic.go:356] "Generic (PLEG): container finished" podID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerID="c3c8402ba52cfd2bc608c16c6c126655230f62d55aaf53cc366a57b3fec471c2" exitCode=0 Nov 02 02:04:03.096912 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.096780 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" Nov 02 02:04:03.096912 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.096834 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" event={"ID":"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94","Type":"ContainerDied","Data":"c3c8402ba52cfd2bc608c16c6c126655230f62d55aaf53cc366a57b3fec471c2"} Nov 02 02:04:03.096912 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.096864 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm" event={"ID":"3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94","Type":"ContainerDied","Data":"0551229e765859f8cfd7af50529c4b689ad7d10b76c7f5e2301e8890470c2902"} Nov 02 02:04:03.107377 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.106878 5044 scope.go:122] "RemoveContainer" containerID="d5bc98f234a9b3c1d46aa172b8418d023df3fd1eadb6ff747a19890a00b06466" Nov 02 02:04:03.107377 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:04:03.107363 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5bc98f234a9b3c1d46aa172b8418d023df3fd1eadb6ff747a19890a00b06466\": container with ID starting with d5bc98f234a9b3c1d46aa172b8418d023df3fd1eadb6ff747a19890a00b06466 not found: ID does not exist" containerID="d5bc98f234a9b3c1d46aa172b8418d023df3fd1eadb6ff747a19890a00b06466" Nov 02 02:04:03.107574 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.107396 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5bc98f234a9b3c1d46aa172b8418d023df3fd1eadb6ff747a19890a00b06466"} err="failed to get container status \"d5bc98f234a9b3c1d46aa172b8418d023df3fd1eadb6ff747a19890a00b06466\": rpc error: code = NotFound desc = could not find container \"d5bc98f234a9b3c1d46aa172b8418d023df3fd1eadb6ff747a19890a00b06466\": container with ID starting with d5bc98f234a9b3c1d46aa172b8418d023df3fd1eadb6ff747a19890a00b06466 not found: ID does not exist" Nov 02 02:04:03.107574 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.107446 5044 scope.go:122] "RemoveContainer" containerID="c3c8402ba52cfd2bc608c16c6c126655230f62d55aaf53cc366a57b3fec471c2" Nov 02 02:04:03.117357 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.117339 5044 scope.go:122] "RemoveContainer" containerID="c3c8402ba52cfd2bc608c16c6c126655230f62d55aaf53cc366a57b3fec471c2" Nov 02 02:04:03.117646 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:04:03.117627 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3c8402ba52cfd2bc608c16c6c126655230f62d55aaf53cc366a57b3fec471c2\": container with ID starting with c3c8402ba52cfd2bc608c16c6c126655230f62d55aaf53cc366a57b3fec471c2 not found: ID does not exist" containerID="c3c8402ba52cfd2bc608c16c6c126655230f62d55aaf53cc366a57b3fec471c2" Nov 02 02:04:03.117761 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.117654 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3c8402ba52cfd2bc608c16c6c126655230f62d55aaf53cc366a57b3fec471c2"} err="failed to get container status \"c3c8402ba52cfd2bc608c16c6c126655230f62d55aaf53cc366a57b3fec471c2\": rpc error: code = NotFound desc = could not find container \"c3c8402ba52cfd2bc608c16c6c126655230f62d55aaf53cc366a57b3fec471c2\": container with ID starting with c3c8402ba52cfd2bc608c16c6c126655230f62d55aaf53cc366a57b3fec471c2 not found: ID does not exist" Nov 02 02:04:03.161637 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.161603 5044 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-tmp-dir\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:03.161637 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.161632 5044 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-home\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:03.161637 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.161644 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s2zd\" (UniqueName: \"kubernetes.io/projected/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-kube-api-access-8s2zd\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:03.161992 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.161654 5044 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-home\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:03.161992 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.161663 5044 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-dshm\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:03.161992 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.161704 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d728t\" (UniqueName: \"kubernetes.io/projected/4e539f99-221e-4050-87f3-a7bd2d1fbb71-kube-api-access-d728t\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:03.161992 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.161714 5044 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-tls-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:03.161992 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.161722 5044 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94-model-cache\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:03.161992 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.161730 5044 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-model-cache\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:03.161992 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.161740 5044 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-tmp-dir\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:03.161992 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.161749 5044 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e539f99-221e-4050-87f3-a7bd2d1fbb71-dshm\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:03.161992 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.161757 5044 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e539f99-221e-4050-87f3-a7bd2d1fbb71-tls-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:03.256572 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.256530 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26"] Nov 02 02:04:03.261036 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.261013 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/premium-w1-f276c3-kserve-75c9f975b8-z2g26"] Nov 02 02:04:03.373747 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.373647 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm"] Nov 02 02:04:03.377425 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.377399 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f276c3/simulator-w1-f276c3-kserve-574fb98789-7m7bm"] Nov 02 02:04:03.452605 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.452575 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" path="/var/lib/kubelet/pods/3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94/volumes" Nov 02 02:04:03.453231 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:03.453212 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" path="/var/lib/kubelet/pods/4e539f99-221e-4050-87f3-a7bd2d1fbb71/volumes" Nov 02 02:04:06.215431 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:06.215399 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:06.357456 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:06.357421 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:06.558061 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:06.558034 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:08.214054 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:08.214016 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:08.358864 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:08.358827 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:09.195899 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:09.195864 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2" podUID="cfeb96d5-3e36-4855-b2e6-7cf5311d131b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 02:04:11.280050 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:11.280013 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:15.408724 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:15.408686 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p"] Nov 02 02:04:15.409195 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:15.408901 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" podUID="35a2761d-bbbb-4614-8d91-a2466216e37e" containerName="maas-api" containerID="cri-o://d07e2cbe7924444fdec8c52ed7d49152a823df76f402739a1518aa11a9d30e0b" gracePeriod=30 Nov 02 02:04:15.436477 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:15.436448 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:15.588026 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:15.587947 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:15.705168 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:15.705141 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" Nov 02 02:04:15.816694 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:15.813304 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35a2761d-bbbb-4614-8d91-a2466216e37e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a2761d-bbbb-4614-8d91-a2466216e37e-maas-api-tls\") pod \"35a2761d-bbbb-4614-8d91-a2466216e37e\" (UID: \"35a2761d-bbbb-4614-8d91-a2466216e37e\") " Nov 02 02:04:15.816694 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:15.813895 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35a2761d-bbbb-4614-8d91-a2466216e37e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a2761d-bbbb-4614-8d91-a2466216e37e-openshift-service-ca\") pod \"35a2761d-bbbb-4614-8d91-a2466216e37e\" (UID: \"35a2761d-bbbb-4614-8d91-a2466216e37e\") " Nov 02 02:04:15.816694 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:15.814027 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a2761d-bbbb-4614-8d91-a2466216e37e-kube-api-access-nnzzn\" (UniqueName: \"kubernetes.io/projected/35a2761d-bbbb-4614-8d91-a2466216e37e-kube-api-access-nnzzn\") pod \"35a2761d-bbbb-4614-8d91-a2466216e37e\" (UID: \"35a2761d-bbbb-4614-8d91-a2466216e37e\") " Nov 02 02:04:15.816694 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:15.814248 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35a2761d-bbbb-4614-8d91-a2466216e37e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a2761d-bbbb-4614-8d91-a2466216e37e-metrics-certs\") pod \"35a2761d-bbbb-4614-8d91-a2466216e37e\" (UID: \"35a2761d-bbbb-4614-8d91-a2466216e37e\") " Nov 02 02:04:15.817595 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:15.817316 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a2761d-bbbb-4614-8d91-a2466216e37e-openshift-service-ca" pod "35a2761d-bbbb-4614-8d91-a2466216e37e" (UID: "35a2761d-bbbb-4614-8d91-a2466216e37e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:04:15.856031 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:15.852848 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a2761d-bbbb-4614-8d91-a2466216e37e-maas-api-tls" pod "35a2761d-bbbb-4614-8d91-a2466216e37e" (UID: "35a2761d-bbbb-4614-8d91-a2466216e37e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:04:15.856031 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:15.852985 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a2761d-bbbb-4614-8d91-a2466216e37e-kube-api-access-nnzzn" pod "35a2761d-bbbb-4614-8d91-a2466216e37e" (UID: "35a2761d-bbbb-4614-8d91-a2466216e37e"). InnerVolumeSpecName "kube-api-access-nnzzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:04:15.856031 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:15.855293 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a2761d-bbbb-4614-8d91-a2466216e37e-metrics-certs" pod "35a2761d-bbbb-4614-8d91-a2466216e37e" (UID: "35a2761d-bbbb-4614-8d91-a2466216e37e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:04:15.915985 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:15.915954 5044 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a2761d-bbbb-4614-8d91-a2466216e37e-openshift-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:15.915985 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:15.915980 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnzzn\" (UniqueName: \"kubernetes.io/projected/35a2761d-bbbb-4614-8d91-a2466216e37e-kube-api-access-nnzzn\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:15.915985 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:15.915990 5044 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a2761d-bbbb-4614-8d91-a2466216e37e-metrics-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:15.916227 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:15.916000 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35a2761d-bbbb-4614-8d91-a2466216e37e-maas-api-tls\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:16.178760 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:16.178716 5044 generic.go:356] "Generic (PLEG): container finished" podID="35a2761d-bbbb-4614-8d91-a2466216e37e" containerID="d07e2cbe7924444fdec8c52ed7d49152a823df76f402739a1518aa11a9d30e0b" exitCode=0 Nov 02 02:04:16.178949 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:16.178763 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" Nov 02 02:04:16.178949 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:16.178805 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" event={"ID":"35a2761d-bbbb-4614-8d91-a2466216e37e","Type":"ContainerDied","Data":"d07e2cbe7924444fdec8c52ed7d49152a823df76f402739a1518aa11a9d30e0b"} Nov 02 02:04:16.178949 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:16.178876 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p" event={"ID":"35a2761d-bbbb-4614-8d91-a2466216e37e","Type":"ContainerDied","Data":"7d69d2c36a7df0eb127b2541ca7325c1db0870a6bfbca0c9de4a437c413c874d"} Nov 02 02:04:16.178949 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:16.178908 5044 scope.go:122] "RemoveContainer" containerID="d07e2cbe7924444fdec8c52ed7d49152a823df76f402739a1518aa11a9d30e0b" Nov 02 02:04:16.191624 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:16.191609 5044 scope.go:122] "RemoveContainer" containerID="d07e2cbe7924444fdec8c52ed7d49152a823df76f402739a1518aa11a9d30e0b" Nov 02 02:04:16.191923 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:04:16.191902 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d07e2cbe7924444fdec8c52ed7d49152a823df76f402739a1518aa11a9d30e0b\": container with ID starting with d07e2cbe7924444fdec8c52ed7d49152a823df76f402739a1518aa11a9d30e0b not found: ID does not exist" containerID="d07e2cbe7924444fdec8c52ed7d49152a823df76f402739a1518aa11a9d30e0b" Nov 02 02:04:16.192003 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:16.191932 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07e2cbe7924444fdec8c52ed7d49152a823df76f402739a1518aa11a9d30e0b"} err="failed to get container status \"d07e2cbe7924444fdec8c52ed7d49152a823df76f402739a1518aa11a9d30e0b\": rpc error: code = NotFound desc = could not find container \"d07e2cbe7924444fdec8c52ed7d49152a823df76f402739a1518aa11a9d30e0b\": container with ID starting with d07e2cbe7924444fdec8c52ed7d49152a823df76f402739a1518aa11a9d30e0b not found: ID does not exist" Nov 02 02:04:16.353267 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:16.353229 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p"] Nov 02 02:04:16.405650 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:16.405614 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b71ba5-6dcf586695-mwv8p"] Nov 02 02:04:16.941082 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:16.941052 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:17.097865 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:17.097827 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:17.352774 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:17.352740 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:17.468735 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:17.468702 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35a2761d-bbbb-4614-8d91-a2466216e37e" path="/var/lib/kubelet/pods/35a2761d-bbbb-4614-8d91-a2466216e37e/volumes" Nov 02 02:04:17.498572 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:17.498542 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:18.006829 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:18.006796 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:19.196102 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:19.196062 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2" podUID="cfeb96d5-3e36-4855-b2e6-7cf5311d131b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 02:04:21.949463 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:21.949420 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:23.141986 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:23.141961 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2" Nov 02 02:04:23.227070 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:23.227035 5044 generic.go:356] "Generic (PLEG): container finished" podID="cfeb96d5-3e36-4855-b2e6-7cf5311d131b" containerID="ec0fe4eab8143f04a4560c06c856130318149f1fa4cb028c9087632d10653123" exitCode=0 Nov 02 02:04:23.227262 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:23.227086 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2" Nov 02 02:04:23.227262 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:23.227123 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2" event={"ID":"cfeb96d5-3e36-4855-b2e6-7cf5311d131b","Type":"ContainerDied","Data":"ec0fe4eab8143f04a4560c06c856130318149f1fa4cb028c9087632d10653123"} Nov 02 02:04:23.227262 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:23.227191 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2" event={"ID":"cfeb96d5-3e36-4855-b2e6-7cf5311d131b","Type":"ContainerDied","Data":"c8427eec97af96388c52f1ff5581319cc14a211fc5e736b88b2a66df4d80fd05"} Nov 02 02:04:23.227262 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:23.227222 5044 scope.go:122] "RemoveContainer" containerID="ec0fe4eab8143f04a4560c06c856130318149f1fa4cb028c9087632d10653123" Nov 02 02:04:23.236720 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:23.236701 5044 scope.go:122] "RemoveContainer" containerID="ec0fe4eab8143f04a4560c06c856130318149f1fa4cb028c9087632d10653123" Nov 02 02:04:23.236997 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:04:23.236977 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec0fe4eab8143f04a4560c06c856130318149f1fa4cb028c9087632d10653123\": container with ID starting with ec0fe4eab8143f04a4560c06c856130318149f1fa4cb028c9087632d10653123 not found: ID does not exist" containerID="ec0fe4eab8143f04a4560c06c856130318149f1fa4cb028c9087632d10653123" Nov 02 02:04:23.237059 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:23.237005 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec0fe4eab8143f04a4560c06c856130318149f1fa4cb028c9087632d10653123"} err="failed to get container status \"ec0fe4eab8143f04a4560c06c856130318149f1fa4cb028c9087632d10653123\": rpc error: code = NotFound desc = could not find container \"ec0fe4eab8143f04a4560c06c856130318149f1fa4cb028c9087632d10653123\": container with ID starting with ec0fe4eab8143f04a4560c06c856130318149f1fa4cb028c9087632d10653123 not found: ID does not exist" Nov 02 02:04:23.329137 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:23.329096 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfeb96d5-3e36-4855-b2e6-7cf5311d131b-kube-api-access-pd8nt\" (UniqueName: \"kubernetes.io/projected/cfeb96d5-3e36-4855-b2e6-7cf5311d131b-kube-api-access-pd8nt\") pod \"cfeb96d5-3e36-4855-b2e6-7cf5311d131b\" (UID: \"cfeb96d5-3e36-4855-b2e6-7cf5311d131b\") " Nov 02 02:04:23.329330 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:23.329225 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfeb96d5-3e36-4855-b2e6-7cf5311d131b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfeb96d5-3e36-4855-b2e6-7cf5311d131b-plugins-config-volume\") pod \"cfeb96d5-3e36-4855-b2e6-7cf5311d131b\" (UID: \"cfeb96d5-3e36-4855-b2e6-7cf5311d131b\") " Nov 02 02:04:23.329625 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:23.329599 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfeb96d5-3e36-4855-b2e6-7cf5311d131b-plugins-config-volume" pod "cfeb96d5-3e36-4855-b2e6-7cf5311d131b" (UID: "cfeb96d5-3e36-4855-b2e6-7cf5311d131b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:04:23.331315 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:23.331292 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfeb96d5-3e36-4855-b2e6-7cf5311d131b-kube-api-access-pd8nt" pod "cfeb96d5-3e36-4855-b2e6-7cf5311d131b" (UID: "cfeb96d5-3e36-4855-b2e6-7cf5311d131b"). InnerVolumeSpecName "kube-api-access-pd8nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:04:23.430980 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:23.430902 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd8nt\" (UniqueName: \"kubernetes.io/projected/cfeb96d5-3e36-4855-b2e6-7cf5311d131b-kube-api-access-pd8nt\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:23.430980 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:23.430936 5044 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfeb96d5-3e36-4855-b2e6-7cf5311d131b-plugins-config-volume\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:23.645969 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:23.645926 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2"] Nov 02 02:04:23.696841 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:23.696770 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f276c3-68cc9594c-xl6f2"] Nov 02 02:04:23.712166 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:23.712138 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:23.937584 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:23.937550 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:24.139811 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:24.139773 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:24.173555 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:04:24.173523 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:04:24.617198 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:24.617159 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:25.115578 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:25.115546 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:25.255451 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:25.255414 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:25.455152 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:25.455070 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfeb96d5-3e36-4855-b2e6-7cf5311d131b" path="/var/lib/kubelet/pods/cfeb96d5-3e36-4855-b2e6-7cf5311d131b/volumes" Nov 02 02:04:25.840100 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:25.840067 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:26.006490 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:26.006458 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:26.214605 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:26.214524 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:26.218136 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:26.218105 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g"] Nov 02 02:04:26.218373 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:26.218336 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" podUID="2320a74e-3e24-4633-8d10-807bcafda325" containerName="istio-proxy" containerID="cri-o://6b7fd1c534894d74d48fa13febedbec7b82d5d6de4e03749b4bdb806adb6f038" gracePeriod=30 Nov 02 02:04:26.364897 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:26.364864 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:29.366969 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:29.366938 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:30.646043 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:30.646015 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:30.846303 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:30.846269 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:31.031244 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.031218 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz"] Nov 02 02:04:31.031452 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.031428 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" podUID="dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" containerName="istio-proxy" containerID="cri-o://cadf401c0c25ae7a51a1b148632572bcb9e56bdbfcee8d6deaffe37721841555" gracePeriod=30 Nov 02 02:04:31.045483 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.045418 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:31.480147 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.480119 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 02:04:31.523505 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.523477 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2320a74e-3e24-4633-8d10-807bcafda325-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2320a74e-3e24-4633-8d10-807bcafda325-istio-podinfo\") pod \"2320a74e-3e24-4633-8d10-807bcafda325\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " Nov 02 02:04:31.523636 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.523555 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2320a74e-3e24-4633-8d10-807bcafda325-istio-token\" (UniqueName: \"kubernetes.io/projected/2320a74e-3e24-4633-8d10-807bcafda325-istio-token\") pod \"2320a74e-3e24-4633-8d10-807bcafda325\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " Nov 02 02:04:31.523636 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.523586 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-workload-socket\") pod \"2320a74e-3e24-4633-8d10-807bcafda325\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " Nov 02 02:04:31.523636 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.523628 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-workload-certs\") pod \"2320a74e-3e24-4633-8d10-807bcafda325\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " Nov 02 02:04:31.523853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.523710 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2320a74e-3e24-4633-8d10-807bcafda325-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2320a74e-3e24-4633-8d10-807bcafda325-istiod-ca-cert\") pod \"2320a74e-3e24-4633-8d10-807bcafda325\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " Nov 02 02:04:31.523853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.523751 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-credential-socket\") pod \"2320a74e-3e24-4633-8d10-807bcafda325\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " Nov 02 02:04:31.523853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.523791 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-istio-envoy\") pod \"2320a74e-3e24-4633-8d10-807bcafda325\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " Nov 02 02:04:31.524022 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.523905 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2320a74e-3e24-4633-8d10-807bcafda325-kube-api-access-jrl22\" (UniqueName: \"kubernetes.io/projected/2320a74e-3e24-4633-8d10-807bcafda325-kube-api-access-jrl22\") pod \"2320a74e-3e24-4633-8d10-807bcafda325\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " Nov 02 02:04:31.524022 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.523931 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-workload-certs" pod "2320a74e-3e24-4633-8d10-807bcafda325" (UID: "2320a74e-3e24-4633-8d10-807bcafda325"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:04:31.524022 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.523942 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-workload-socket" pod "2320a74e-3e24-4633-8d10-807bcafda325" (UID: "2320a74e-3e24-4633-8d10-807bcafda325"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:04:31.524022 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.524005 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-istio-data\") pod \"2320a74e-3e24-4633-8d10-807bcafda325\" (UID: \"2320a74e-3e24-4633-8d10-807bcafda325\") " Nov 02 02:04:31.524253 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.524155 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-credential-socket" pod "2320a74e-3e24-4633-8d10-807bcafda325" (UID: "2320a74e-3e24-4633-8d10-807bcafda325"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:04:31.524330 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.524251 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-istio-data" pod "2320a74e-3e24-4633-8d10-807bcafda325" (UID: "2320a74e-3e24-4633-8d10-807bcafda325"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:04:31.524390 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.524369 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2320a74e-3e24-4633-8d10-807bcafda325-istiod-ca-cert" pod "2320a74e-3e24-4633-8d10-807bcafda325" (UID: "2320a74e-3e24-4633-8d10-807bcafda325"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:04:31.524853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.524744 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:31.524853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.524766 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:31.524853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.524780 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2320a74e-3e24-4633-8d10-807bcafda325-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:31.524853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.524793 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:31.524853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.524807 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:31.525981 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.525952 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2320a74e-3e24-4633-8d10-807bcafda325-istio-podinfo" pod "2320a74e-3e24-4633-8d10-807bcafda325" (UID: "2320a74e-3e24-4633-8d10-807bcafda325"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 02:04:31.526117 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.526018 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2320a74e-3e24-4633-8d10-807bcafda325-istio-token" pod "2320a74e-3e24-4633-8d10-807bcafda325" (UID: "2320a74e-3e24-4633-8d10-807bcafda325"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:04:31.526360 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.526339 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-istio-envoy" pod "2320a74e-3e24-4633-8d10-807bcafda325" (UID: "2320a74e-3e24-4633-8d10-807bcafda325"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:04:31.526470 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.526373 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2320a74e-3e24-4633-8d10-807bcafda325-kube-api-access-jrl22" pod "2320a74e-3e24-4633-8d10-807bcafda325" (UID: "2320a74e-3e24-4633-8d10-807bcafda325"). InnerVolumeSpecName "kube-api-access-jrl22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:04:31.625876 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.625801 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2320a74e-3e24-4633-8d10-807bcafda325-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:31.625876 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.625829 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2320a74e-3e24-4633-8d10-807bcafda325-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:31.625876 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.625840 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrl22\" (UniqueName: \"kubernetes.io/projected/2320a74e-3e24-4633-8d10-807bcafda325-kube-api-access-jrl22\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:31.625876 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:31.625849 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2320a74e-3e24-4633-8d10-807bcafda325-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:32.295048 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:32.295010 5044 generic.go:356] "Generic (PLEG): container finished" podID="2320a74e-3e24-4633-8d10-807bcafda325" containerID="6b7fd1c534894d74d48fa13febedbec7b82d5d6de4e03749b4bdb806adb6f038" exitCode=0 Nov 02 02:04:32.295533 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:32.295057 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" event={"ID":"2320a74e-3e24-4633-8d10-807bcafda325","Type":"ContainerDied","Data":"6b7fd1c534894d74d48fa13febedbec7b82d5d6de4e03749b4bdb806adb6f038"} Nov 02 02:04:32.295533 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:32.295113 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" event={"ID":"2320a74e-3e24-4633-8d10-807bcafda325","Type":"ContainerDied","Data":"1c7d89690bcb1ddc2028f4213fba253f03e30c33f81d039def8a37f50a441d13"} Nov 02 02:04:32.295533 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:32.295129 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g" Nov 02 02:04:32.295533 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:32.295141 5044 scope.go:122] "RemoveContainer" containerID="6b7fd1c534894d74d48fa13febedbec7b82d5d6de4e03749b4bdb806adb6f038" Nov 02 02:04:32.306739 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:32.306717 5044 scope.go:122] "RemoveContainer" containerID="6b7fd1c534894d74d48fa13febedbec7b82d5d6de4e03749b4bdb806adb6f038" Nov 02 02:04:32.307038 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:04:32.307019 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b7fd1c534894d74d48fa13febedbec7b82d5d6de4e03749b4bdb806adb6f038\": container with ID starting with 6b7fd1c534894d74d48fa13febedbec7b82d5d6de4e03749b4bdb806adb6f038 not found: ID does not exist" containerID="6b7fd1c534894d74d48fa13febedbec7b82d5d6de4e03749b4bdb806adb6f038" Nov 02 02:04:32.307127 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:32.307048 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b7fd1c534894d74d48fa13febedbec7b82d5d6de4e03749b4bdb806adb6f038"} err="failed to get container status \"6b7fd1c534894d74d48fa13febedbec7b82d5d6de4e03749b4bdb806adb6f038\": rpc error: code = NotFound desc = could not find container \"6b7fd1c534894d74d48fa13febedbec7b82d5d6de4e03749b4bdb806adb6f038\": container with ID starting with 6b7fd1c534894d74d48fa13febedbec7b82d5d6de4e03749b4bdb806adb6f038 not found: ID does not exist" Nov 02 02:04:32.370577 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:32.370530 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:32.465772 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:32.465738 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g"] Nov 02 02:04:32.516556 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:32.516525 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b71ba5-openshift-default-846d4d49cc-vqm2g"] Nov 02 02:04:33.453902 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.453868 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2320a74e-3e24-4633-8d10-807bcafda325" path="/var/lib/kubelet/pods/2320a74e-3e24-4633-8d10-807bcafda325/volumes" Nov 02 02:04:33.905518 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.905490 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf"] Nov 02 02:04:33.906027 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906013 5044 cpu_manager.go:414] "containerMap: removing container" podUID="cfeb96d5-3e36-4855-b2e6-7cf5311d131b" containerName="payload-processing" Nov 02 02:04:33.906077 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906029 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfeb96d5-3e36-4855-b2e6-7cf5311d131b" containerName="payload-processing" Nov 02 02:04:33.906077 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906042 5044 cpu_manager.go:414] "containerMap: removing container" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" Nov 02 02:04:33.906077 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906047 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" Nov 02 02:04:33.906077 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906062 5044 cpu_manager.go:414] "containerMap: removing container" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" Nov 02 02:04:33.906077 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906067 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" Nov 02 02:04:33.906077 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906075 5044 cpu_manager.go:414] "containerMap: removing container" podUID="35a2761d-bbbb-4614-8d91-a2466216e37e" containerName="maas-api" Nov 02 02:04:33.906269 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906080 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a2761d-bbbb-4614-8d91-a2466216e37e" containerName="maas-api" Nov 02 02:04:33.906269 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906100 5044 cpu_manager.go:414] "containerMap: removing container" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" Nov 02 02:04:33.906269 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906105 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" Nov 02 02:04:33.906269 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906122 5044 cpu_manager.go:414] "containerMap: removing container" podUID="2320a74e-3e24-4633-8d10-807bcafda325" containerName="istio-proxy" Nov 02 02:04:33.906269 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906127 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2320a74e-3e24-4633-8d10-807bcafda325" containerName="istio-proxy" Nov 02 02:04:33.906269 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906134 5044 cpu_manager.go:414] "containerMap: removing container" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" Nov 02 02:04:33.906269 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906140 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" Nov 02 02:04:33.906269 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906221 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a1c8fde-4ec7-433a-9f68-0470dee95c46" containerName="main" Nov 02 02:04:33.906269 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906234 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="2320a74e-3e24-4633-8d10-807bcafda325" containerName="istio-proxy" Nov 02 02:04:33.906269 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906241 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="70609eac-cd4a-4e96-84b0-536dccf3a715" containerName="main" Nov 02 02:04:33.906269 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906247 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfeb96d5-3e36-4855-b2e6-7cf5311d131b" containerName="payload-processing" Nov 02 02:04:33.906269 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906251 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1661ef-7b1a-48fe-9514-e1ffe7ba7a94" containerName="main" Nov 02 02:04:33.906269 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906257 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e539f99-221e-4050-87f3-a7bd2d1fbb71" containerName="main" Nov 02 02:04:33.906269 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.906263 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a2761d-bbbb-4614-8d91-a2466216e37e" containerName="maas-api" Nov 02 02:04:33.910939 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.910924 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" Nov 02 02:04:33.953090 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.953054 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0186e1-b5ca-4475-b1bd-0003168fa850-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf\" (UID: \"6a0186e1-b5ca-4475-b1bd-0003168fa850\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" Nov 02 02:04:33.953248 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.953191 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a0186e1-b5ca-4475-b1bd-0003168fa850-maas-api-tls\") pod \"maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf\" (UID: \"6a0186e1-b5ca-4475-b1bd-0003168fa850\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" Nov 02 02:04:33.953322 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.953277 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a0186e1-b5ca-4475-b1bd-0003168fa850-metrics-certs\") pod \"maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf\" (UID: \"6a0186e1-b5ca-4475-b1bd-0003168fa850\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" Nov 02 02:04:33.953397 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.953384 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fqrs\" (UniqueName: \"kubernetes.io/projected/6a0186e1-b5ca-4475-b1bd-0003168fa850-kube-api-access-9fqrs\") pod \"maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf\" (UID: \"6a0186e1-b5ca-4475-b1bd-0003168fa850\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" Nov 02 02:04:33.969429 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:33.969398 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d0fdb9-serving-cert\"" Nov 02 02:04:34.023137 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:34.023109 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf"] Nov 02 02:04:34.054300 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:34.054262 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a0186e1-b5ca-4475-b1bd-0003168fa850-maas-api-tls\") pod \"maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf\" (UID: \"6a0186e1-b5ca-4475-b1bd-0003168fa850\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" Nov 02 02:04:34.054300 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:34.054314 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a0186e1-b5ca-4475-b1bd-0003168fa850-metrics-certs\") pod \"maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf\" (UID: \"6a0186e1-b5ca-4475-b1bd-0003168fa850\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" Nov 02 02:04:34.054550 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:34.054394 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fqrs\" (UniqueName: \"kubernetes.io/projected/6a0186e1-b5ca-4475-b1bd-0003168fa850-kube-api-access-9fqrs\") pod \"maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf\" (UID: \"6a0186e1-b5ca-4475-b1bd-0003168fa850\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" Nov 02 02:04:34.054550 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:34.054453 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0186e1-b5ca-4475-b1bd-0003168fa850-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf\" (UID: \"6a0186e1-b5ca-4475-b1bd-0003168fa850\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" Nov 02 02:04:34.054550 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:04:34.054466 5044 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-serving-cert: secret "maas-api-e2e-inf-a-d0fdb9-serving-cert" not found Nov 02 02:04:34.054550 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:04:34.054539 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a0186e1-b5ca-4475-b1bd-0003168fa850-maas-api-tls podName:6a0186e1-b5ca-4475-b1bd-0003168fa850 nodeName:}" failed. No retries permitted until 2026-11-02 02:04:34.55451814 +0000 UTC m=+2391.225770801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a0186e1-b5ca-4475-b1bd-0003168fa850-maas-api-tls") pod "maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" (UID: "6a0186e1-b5ca-4475-b1bd-0003168fa850") : secret "maas-api-e2e-inf-a-d0fdb9-serving-cert" not found Nov 02 02:04:34.055580 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:34.055551 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0186e1-b5ca-4475-b1bd-0003168fa850-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf\" (UID: \"6a0186e1-b5ca-4475-b1bd-0003168fa850\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" Nov 02 02:04:34.057549 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:34.057525 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a0186e1-b5ca-4475-b1bd-0003168fa850-metrics-certs\") pod \"maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf\" (UID: \"6a0186e1-b5ca-4475-b1bd-0003168fa850\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" Nov 02 02:04:34.101297 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:34.101270 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:34.117320 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:34.117295 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fqrs\" (UniqueName: \"kubernetes.io/projected/6a0186e1-b5ca-4475-b1bd-0003168fa850-kube-api-access-9fqrs\") pod \"maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf\" (UID: \"6a0186e1-b5ca-4475-b1bd-0003168fa850\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" Nov 02 02:04:34.259057 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:34.259027 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:34.406453 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:34.406414 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:34.563459 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:34.563375 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a0186e1-b5ca-4475-b1bd-0003168fa850-maas-api-tls\") pod \"maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf\" (UID: \"6a0186e1-b5ca-4475-b1bd-0003168fa850\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" Nov 02 02:04:34.565907 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:34.565880 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a0186e1-b5ca-4475-b1bd-0003168fa850-maas-api-tls\") pod \"maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf\" (UID: \"6a0186e1-b5ca-4475-b1bd-0003168fa850\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" Nov 02 02:04:34.824092 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:34.824014 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" Nov 02 02:04:35.204358 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:35.204330 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf"] Nov 02 02:04:35.205044 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:04:35.205018 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0186e1_b5ca_4475_b1bd_0003168fa850.slice/crio-fa73b54226b98541d51f86ecf750f2516730af822f844dfdabbb7115df82dd88 WatchSource:0}: Error finding container fa73b54226b98541d51f86ecf750f2516730af822f844dfdabbb7115df82dd88: Status 404 returned error can't find the container with id fa73b54226b98541d51f86ecf750f2516730af822f844dfdabbb7115df82dd88 Nov 02 02:04:35.319474 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:35.319438 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" event={"ID":"6a0186e1-b5ca-4475-b1bd-0003168fa850","Type":"ContainerStarted","Data":"fa73b54226b98541d51f86ecf750f2516730af822f844dfdabbb7115df82dd88"} Nov 02 02:04:36.302925 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.302897 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 02:04:36.326134 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.326108 5044 generic.go:356] "Generic (PLEG): container finished" podID="dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" containerID="cadf401c0c25ae7a51a1b148632572bcb9e56bdbfcee8d6deaffe37721841555" exitCode=0 Nov 02 02:04:36.326283 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.326148 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" Nov 02 02:04:36.326283 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.326189 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" event={"ID":"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258","Type":"ContainerDied","Data":"cadf401c0c25ae7a51a1b148632572bcb9e56bdbfcee8d6deaffe37721841555"} Nov 02 02:04:36.326283 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.326258 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz" event={"ID":"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258","Type":"ContainerDied","Data":"e9b00f89e1406221804ab910363fae17288425b92775d995d2d8bd35e0c3c1cf"} Nov 02 02:04:36.326417 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.326292 5044 scope.go:122] "RemoveContainer" containerID="cadf401c0c25ae7a51a1b148632572bcb9e56bdbfcee8d6deaffe37721841555" Nov 02 02:04:36.327896 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.327875 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" event={"ID":"6a0186e1-b5ca-4475-b1bd-0003168fa850","Type":"ContainerStarted","Data":"221b434a0fecf0566722b1eac8d2cf2a7bcc540915e69b343687f634dabe5f11"} Nov 02 02:04:36.328029 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.327919 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" Nov 02 02:04:36.336284 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.336268 5044 scope.go:122] "RemoveContainer" containerID="cadf401c0c25ae7a51a1b148632572bcb9e56bdbfcee8d6deaffe37721841555" Nov 02 02:04:36.336548 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:04:36.336527 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cadf401c0c25ae7a51a1b148632572bcb9e56bdbfcee8d6deaffe37721841555\": container with ID starting with cadf401c0c25ae7a51a1b148632572bcb9e56bdbfcee8d6deaffe37721841555 not found: ID does not exist" containerID="cadf401c0c25ae7a51a1b148632572bcb9e56bdbfcee8d6deaffe37721841555" Nov 02 02:04:36.336600 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.336557 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cadf401c0c25ae7a51a1b148632572bcb9e56bdbfcee8d6deaffe37721841555"} err="failed to get container status \"cadf401c0c25ae7a51a1b148632572bcb9e56bdbfcee8d6deaffe37721841555\": rpc error: code = NotFound desc = could not find container \"cadf401c0c25ae7a51a1b148632572bcb9e56bdbfcee8d6deaffe37721841555\": container with ID starting with cadf401c0c25ae7a51a1b148632572bcb9e56bdbfcee8d6deaffe37721841555 not found: ID does not exist" Nov 02 02:04:36.385525 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.385505 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-data\") pod \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " Nov 02 02:04:36.385634 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.385555 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-podinfo\") pod \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " Nov 02 02:04:36.385634 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.385626 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-token\" (UniqueName: \"kubernetes.io/projected/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-token\") pod \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " Nov 02 02:04:36.385805 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.385666 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-envoy\") pod \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " Nov 02 02:04:36.385805 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.385722 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-kube-api-access-9sldl\" (UniqueName: \"kubernetes.io/projected/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-kube-api-access-9sldl\") pod \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " Nov 02 02:04:36.385805 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.385766 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istiod-ca-cert\") pod \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " Nov 02 02:04:36.385963 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.385801 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-data" pod "dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" (UID: "dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:04:36.385963 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.385828 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-workload-certs\") pod \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " Nov 02 02:04:36.385963 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.385859 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-workload-socket\") pod \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " Nov 02 02:04:36.385963 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.385886 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-credential-socket\") pod \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\" (UID: \"dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258\") " Nov 02 02:04:36.386482 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.386459 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:36.386992 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.386970 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-credential-socket" pod "dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" (UID: "dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:04:36.387525 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.387498 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-workload-socket" pod "dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" (UID: "dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:04:36.387622 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.387604 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-workload-certs" pod "dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" (UID: "dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:04:36.387729 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.387666 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istiod-ca-cert" pod "dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" (UID: "dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:04:36.388142 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.388117 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-kube-api-access-9sldl" pod "dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" (UID: "dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258"). InnerVolumeSpecName "kube-api-access-9sldl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:04:36.388300 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.388281 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-podinfo" pod "dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" (UID: "dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 02:04:36.388376 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.388356 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-token" pod "dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" (UID: "dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:04:36.388706 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.388655 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-envoy" pod "dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" (UID: "dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:04:36.487352 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.487320 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:36.487352 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.487355 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:36.487549 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.487371 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:36.487549 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.487388 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sldl\" (UniqueName: \"kubernetes.io/projected/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-kube-api-access-9sldl\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:36.487549 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.487403 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:36.487549 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.487418 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:36.487549 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.487433 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:36.487549 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.487447 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:36.600998 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.600895 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" podStartSLOduration=2.806319019 podStartE2EDuration="3.600880056s" totalImagesPullingTime="794.561037ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:04:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:04:36.59825075 +0000 UTC m=+2393.269503431" watchObservedRunningTime="2026-11-02 02:04:36.600880056 +0000 UTC m=+2393.272132736" Nov 02 02:04:36.767429 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.767389 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz"] Nov 02 02:04:36.772477 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:36.772451 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-187478-openshift-default-7fd8468599-frzvz"] Nov 02 02:04:37.452377 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:37.452339 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" path="/var/lib/kubelet/pods/dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258/volumes" Nov 02 02:04:38.310168 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:38.310138 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:39.045804 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:39.045769 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:39.687142 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:39.687104 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:39.887295 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:39.887241 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:40.101301 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:40.101274 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq"] Nov 02 02:04:40.101823 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:40.101808 5044 cpu_manager.go:414] "containerMap: removing container" podUID="dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" containerName="istio-proxy" Nov 02 02:04:40.101866 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:40.101824 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" containerName="istio-proxy" Nov 02 02:04:40.101930 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:40.101921 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="dace65e4-e8b0-4a4b-84cf-b6bc1e0e7258" containerName="istio-proxy" Nov 02 02:04:40.106070 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:40.106049 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq" Nov 02 02:04:40.161790 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:40.161766 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d0fdb9\"" Nov 02 02:04:40.219993 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:40.219954 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq"] Nov 02 02:04:40.234908 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:40.234873 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlntl\" (UniqueName: \"kubernetes.io/projected/dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8-kube-api-access-tlntl\") pod \"payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq\" (UID: \"dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq" Nov 02 02:04:40.235093 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:40.234987 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq\" (UID: \"dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq" Nov 02 02:04:40.335812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:40.335773 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlntl\" (UniqueName: \"kubernetes.io/projected/dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8-kube-api-access-tlntl\") pod \"payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq\" (UID: \"dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq" Nov 02 02:04:40.336005 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:40.335853 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq\" (UID: \"dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq" Nov 02 02:04:40.336538 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:40.336512 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq\" (UID: \"dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq" Nov 02 02:04:40.398916 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:40.398842 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlntl\" (UniqueName: \"kubernetes.io/projected/dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8-kube-api-access-tlntl\") pod \"payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq\" (UID: \"dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq" Nov 02 02:04:40.421176 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:40.421146 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq" Nov 02 02:04:40.802618 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:40.802590 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq"] Nov 02 02:04:40.804554 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:04:40.804522 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae5f42f_91d9_49f6_8f7f_84dbffc0f1a8.slice/crio-dfcbf8f30deffd59e230bca909300319ca13a20da298fe3d7fc3e795bf2afb8f WatchSource:0}: Error finding container dfcbf8f30deffd59e230bca909300319ca13a20da298fe3d7fc3e795bf2afb8f: Status 404 returned error can't find the container with id dfcbf8f30deffd59e230bca909300319ca13a20da298fe3d7fc3e795bf2afb8f Nov 02 02:04:41.360607 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:41.360571 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq" event={"ID":"dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8","Type":"ContainerStarted","Data":"cc3c3898bb35306d643b460c592251a04bbc9d256e1bf7bbb7652bb952c9f78a"} Nov 02 02:04:41.361028 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:41.360636 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq" event={"ID":"dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8","Type":"ContainerStarted","Data":"dfcbf8f30deffd59e230bca909300319ca13a20da298fe3d7fc3e795bf2afb8f"} Nov 02 02:04:41.361028 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:41.360665 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq" Nov 02 02:04:41.480360 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:41.480290 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq" podStartSLOduration=1.480270051 podStartE2EDuration="1.480270051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:04:41.479466358 +0000 UTC m=+2398.150719038" watchObservedRunningTime="2026-11-02 02:04:41.480270051 +0000 UTC m=+2398.151522732" Nov 02 02:04:42.340229 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:42.340203 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" Nov 02 02:04:43.765499 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:43.765466 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 02:04:43.771143 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:43.771121 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 02:04:43.935254 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:43.935223 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5"] Nov 02 02:04:43.939302 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:43.939288 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" Nov 02 02:04:43.994961 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:43.994935 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-76e530-serving-cert\"" Nov 02 02:04:44.048768 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:44.048689 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5"] Nov 02 02:04:44.076813 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:44.076779 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvfll\" (UniqueName: \"kubernetes.io/projected/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-kube-api-access-kvfll\") pod \"maas-api-e2e-api-a-76e530-766d49b76f-pgsl5\" (UID: \"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" Nov 02 02:04:44.077109 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:44.077092 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-openshift-service-ca\") pod \"maas-api-e2e-api-a-76e530-766d49b76f-pgsl5\" (UID: \"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" Nov 02 02:04:44.077361 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:44.077331 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-maas-api-tls\") pod \"maas-api-e2e-api-a-76e530-766d49b76f-pgsl5\" (UID: \"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" Nov 02 02:04:44.077474 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:44.077381 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-metrics-certs\") pod \"maas-api-e2e-api-a-76e530-766d49b76f-pgsl5\" (UID: \"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" Nov 02 02:04:44.134430 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:44.134399 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:44.178476 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:44.178442 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvfll\" (UniqueName: \"kubernetes.io/projected/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-kube-api-access-kvfll\") pod \"maas-api-e2e-api-a-76e530-766d49b76f-pgsl5\" (UID: \"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" Nov 02 02:04:44.181180 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:44.181151 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-openshift-service-ca\") pod \"maas-api-e2e-api-a-76e530-766d49b76f-pgsl5\" (UID: \"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" Nov 02 02:04:44.183620 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:44.178532 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-openshift-service-ca\") pod \"maas-api-e2e-api-a-76e530-766d49b76f-pgsl5\" (UID: \"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" Nov 02 02:04:44.184544 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:44.184517 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-maas-api-tls\") pod \"maas-api-e2e-api-a-76e530-766d49b76f-pgsl5\" (UID: \"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" Nov 02 02:04:44.184756 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:44.184735 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-metrics-certs\") pod \"maas-api-e2e-api-a-76e530-766d49b76f-pgsl5\" (UID: \"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" Nov 02 02:04:44.189195 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:44.189159 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-maas-api-tls\") pod \"maas-api-e2e-api-a-76e530-766d49b76f-pgsl5\" (UID: \"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" Nov 02 02:04:44.204042 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:44.203978 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-metrics-certs\") pod \"maas-api-e2e-api-a-76e530-766d49b76f-pgsl5\" (UID: \"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" Nov 02 02:04:44.241455 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:44.241425 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvfll\" (UniqueName: \"kubernetes.io/projected/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-kube-api-access-kvfll\") pod \"maas-api-e2e-api-a-76e530-766d49b76f-pgsl5\" (UID: \"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" Nov 02 02:04:44.253356 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:44.253294 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" Nov 02 02:04:44.296863 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:44.296829 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:44.446183 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:44.446161 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:44.633564 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:44.633533 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5"] Nov 02 02:04:44.634844 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:04:44.634811 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea7ab8f_45cb_4ce3_af7e_896e2e049f0d.slice/crio-69169ec23e823aee774288f0e4e164833501b7d12073495ce641e44edffbb885 WatchSource:0}: Error finding container 69169ec23e823aee774288f0e4e164833501b7d12073495ce641e44edffbb885: Status 404 returned error can't find the container with id 69169ec23e823aee774288f0e4e164833501b7d12073495ce641e44edffbb885 Nov 02 02:04:45.180779 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:45.180731 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:45.331134 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:45.331096 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:45.384891 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:45.384851 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" event={"ID":"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d","Type":"ContainerStarted","Data":"69169ec23e823aee774288f0e4e164833501b7d12073495ce641e44edffbb885"} Nov 02 02:04:46.393586 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:46.393552 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" event={"ID":"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d","Type":"ContainerStarted","Data":"7488323eda2119b38eb132538a3f76c6e5162aba91941fed1b65e6d526e14565"} Nov 02 02:04:46.394090 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:46.393645 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" Nov 02 02:04:46.514943 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:46.514894 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" podStartSLOduration=2.700054594 podStartE2EDuration="3.514881638s" totalImagesPullingTime="814.827044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:04:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:04:46.514235074 +0000 UTC m=+2403.185487756" watchObservedRunningTime="2026-11-02 02:04:46.514881638 +0000 UTC m=+2403.186134318" Nov 02 02:04:46.811842 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:46.811807 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:48.434468 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:48.434435 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5"] Nov 02 02:04:48.434938 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:48.434637 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" podUID="f20c0468-c6b6-4718-b3f2-718c8c044fb9" containerName="maas-api" containerID="cri-o://7bfd0b48706bd64dee09cf5f1a1f2f35e66495704b3401c38457f308c92b2356" gracePeriod=30 Nov 02 02:04:48.684860 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:48.684709 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:48.736468 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:48.736442 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" Nov 02 02:04:48.840690 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:48.840649 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f20c0468-c6b6-4718-b3f2-718c8c044fb9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f20c0468-c6b6-4718-b3f2-718c8c044fb9-maas-api-tls\") pod \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\" (UID: \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\") " Nov 02 02:04:48.840884 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:48.840854 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f20c0468-c6b6-4718-b3f2-718c8c044fb9-kube-api-access-d6txb\" (UniqueName: \"kubernetes.io/projected/f20c0468-c6b6-4718-b3f2-718c8c044fb9-kube-api-access-d6txb\") pod \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\" (UID: \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\") " Nov 02 02:04:48.840953 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:48.840903 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f20c0468-c6b6-4718-b3f2-718c8c044fb9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f20c0468-c6b6-4718-b3f2-718c8c044fb9-metrics-certs\") pod \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\" (UID: \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\") " Nov 02 02:04:48.841011 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:48.840996 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f20c0468-c6b6-4718-b3f2-718c8c044fb9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f20c0468-c6b6-4718-b3f2-718c8c044fb9-openshift-service-ca\") pod \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\" (UID: \"f20c0468-c6b6-4718-b3f2-718c8c044fb9\") " Nov 02 02:04:48.841730 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:48.841660 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20c0468-c6b6-4718-b3f2-718c8c044fb9-openshift-service-ca" pod "f20c0468-c6b6-4718-b3f2-718c8c044fb9" (UID: "f20c0468-c6b6-4718-b3f2-718c8c044fb9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:04:48.845260 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:48.844959 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f20c0468-c6b6-4718-b3f2-718c8c044fb9-maas-api-tls" pod "f20c0468-c6b6-4718-b3f2-718c8c044fb9" (UID: "f20c0468-c6b6-4718-b3f2-718c8c044fb9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:04:48.847374 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:48.846877 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20c0468-c6b6-4718-b3f2-718c8c044fb9-kube-api-access-d6txb" pod "f20c0468-c6b6-4718-b3f2-718c8c044fb9" (UID: "f20c0468-c6b6-4718-b3f2-718c8c044fb9"). InnerVolumeSpecName "kube-api-access-d6txb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:04:48.860940 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:48.860887 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f20c0468-c6b6-4718-b3f2-718c8c044fb9-metrics-certs" pod "f20c0468-c6b6-4718-b3f2-718c8c044fb9" (UID: "f20c0468-c6b6-4718-b3f2-718c8c044fb9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:04:48.942806 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:48.942733 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6txb\" (UniqueName: \"kubernetes.io/projected/f20c0468-c6b6-4718-b3f2-718c8c044fb9-kube-api-access-d6txb\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:48.942806 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:48.942765 5044 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f20c0468-c6b6-4718-b3f2-718c8c044fb9-metrics-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:48.942806 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:48.942775 5044 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f20c0468-c6b6-4718-b3f2-718c8c044fb9-openshift-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:48.942806 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:48.942784 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f20c0468-c6b6-4718-b3f2-718c8c044fb9-maas-api-tls\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:04:49.420744 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:49.420702 5044 generic.go:356] "Generic (PLEG): container finished" podID="f20c0468-c6b6-4718-b3f2-718c8c044fb9" containerID="7bfd0b48706bd64dee09cf5f1a1f2f35e66495704b3401c38457f308c92b2356" exitCode=0 Nov 02 02:04:49.421196 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:49.421170 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" event={"ID":"f20c0468-c6b6-4718-b3f2-718c8c044fb9","Type":"ContainerDied","Data":"7bfd0b48706bd64dee09cf5f1a1f2f35e66495704b3401c38457f308c92b2356"} Nov 02 02:04:49.421365 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:49.421349 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" event={"ID":"f20c0468-c6b6-4718-b3f2-718c8c044fb9","Type":"ContainerDied","Data":"a473341affca3c6e4f6bdd7c2f696bc3a21ddc7d35df33b6f21ef4d7de74f8ff"} Nov 02 02:04:49.421487 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:49.421473 5044 scope.go:122] "RemoveContainer" containerID="7bfd0b48706bd64dee09cf5f1a1f2f35e66495704b3401c38457f308c92b2356" Nov 02 02:04:49.421774 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:49.421755 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" Nov 02 02:04:49.443894 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:49.443512 5044 scope.go:122] "RemoveContainer" containerID="7bfd0b48706bd64dee09cf5f1a1f2f35e66495704b3401c38457f308c92b2356" Nov 02 02:04:49.444517 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:04:49.444251 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bfd0b48706bd64dee09cf5f1a1f2f35e66495704b3401c38457f308c92b2356\": container with ID starting with 7bfd0b48706bd64dee09cf5f1a1f2f35e66495704b3401c38457f308c92b2356 not found: ID does not exist" containerID="7bfd0b48706bd64dee09cf5f1a1f2f35e66495704b3401c38457f308c92b2356" Nov 02 02:04:49.444517 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:49.444287 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bfd0b48706bd64dee09cf5f1a1f2f35e66495704b3401c38457f308c92b2356"} err="failed to get container status \"7bfd0b48706bd64dee09cf5f1a1f2f35e66495704b3401c38457f308c92b2356\": rpc error: code = NotFound desc = could not find container \"7bfd0b48706bd64dee09cf5f1a1f2f35e66495704b3401c38457f308c92b2356\": container with ID starting with 7bfd0b48706bd64dee09cf5f1a1f2f35e66495704b3401c38457f308c92b2356 not found: ID does not exist" Nov 02 02:04:49.602465 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:49.602432 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5"] Nov 02 02:04:49.653776 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:49.653744 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5"] Nov 02 02:04:49.877031 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:49.877001 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:50.051842 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.051814 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:50.203224 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.203130 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:50.288968 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.288927 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42"] Nov 02 02:04:50.289530 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.289510 5044 cpu_manager.go:414] "containerMap: removing container" podUID="f20c0468-c6b6-4718-b3f2-718c8c044fb9" containerName="maas-api" Nov 02 02:04:50.289636 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.289531 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f20c0468-c6b6-4718-b3f2-718c8c044fb9" containerName="maas-api" Nov 02 02:04:50.289739 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.289726 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="f20c0468-c6b6-4718-b3f2-718c8c044fb9" containerName="maas-api" Nov 02 02:04:50.294357 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.294336 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42" Nov 02 02:04:50.348908 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.348879 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-76e530\"" Nov 02 02:04:50.349074 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.349031 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-76e530-dockercfg-q64b8\"" Nov 02 02:04:50.350188 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.350165 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42"] Nov 02 02:04:50.370033 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.369987 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:50.457351 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.457251 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26"] Nov 02 02:04:50.470572 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.463754 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkcrz\" (UniqueName: \"kubernetes.io/projected/85abf3dc-3eea-41cc-bbaf-2d0208adc731-kube-api-access-qkcrz\") pod \"payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42\" (UID: \"85abf3dc-3eea-41cc-bbaf-2d0208adc731\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42" Nov 02 02:04:50.470572 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.470330 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85abf3dc-3eea-41cc-bbaf-2d0208adc731-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42\" (UID: \"85abf3dc-3eea-41cc-bbaf-2d0208adc731\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42" Nov 02 02:04:50.483709 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.475831 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26" Nov 02 02:04:50.512698 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.509545 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26"] Nov 02 02:04:50.516693 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.514692 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:50.571632 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.571595 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85abf3dc-3eea-41cc-bbaf-2d0208adc731-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42\" (UID: \"85abf3dc-3eea-41cc-bbaf-2d0208adc731\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42" Nov 02 02:04:50.571838 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.571728 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-567d8\" (UniqueName: \"kubernetes.io/projected/b1122c44-90ba-41b2-a10a-72e917ae092e-kube-api-access-567d8\") pod \"payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26\" (UID: \"b1122c44-90ba-41b2-a10a-72e917ae092e\") " pod="openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26" Nov 02 02:04:50.571838 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.571815 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkcrz\" (UniqueName: \"kubernetes.io/projected/85abf3dc-3eea-41cc-bbaf-2d0208adc731-kube-api-access-qkcrz\") pod \"payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42\" (UID: \"85abf3dc-3eea-41cc-bbaf-2d0208adc731\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42" Nov 02 02:04:50.571971 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.571852 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1122c44-90ba-41b2-a10a-72e917ae092e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26\" (UID: \"b1122c44-90ba-41b2-a10a-72e917ae092e\") " pod="openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26" Nov 02 02:04:50.572430 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.572407 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85abf3dc-3eea-41cc-bbaf-2d0208adc731-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42\" (UID: \"85abf3dc-3eea-41cc-bbaf-2d0208adc731\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42" Nov 02 02:04:50.634812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.634776 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkcrz\" (UniqueName: \"kubernetes.io/projected/85abf3dc-3eea-41cc-bbaf-2d0208adc731-kube-api-access-qkcrz\") pod \"payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42\" (UID: \"85abf3dc-3eea-41cc-bbaf-2d0208adc731\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42" Nov 02 02:04:50.673159 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.673129 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-567d8\" (UniqueName: \"kubernetes.io/projected/b1122c44-90ba-41b2-a10a-72e917ae092e-kube-api-access-567d8\") pod \"payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26\" (UID: \"b1122c44-90ba-41b2-a10a-72e917ae092e\") " pod="openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26" Nov 02 02:04:50.673333 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.673240 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1122c44-90ba-41b2-a10a-72e917ae092e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26\" (UID: \"b1122c44-90ba-41b2-a10a-72e917ae092e\") " pod="openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26" Nov 02 02:04:50.674197 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.674168 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1122c44-90ba-41b2-a10a-72e917ae092e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26\" (UID: \"b1122c44-90ba-41b2-a10a-72e917ae092e\") " pod="openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26" Nov 02 02:04:50.736470 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.736406 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-567d8\" (UniqueName: \"kubernetes.io/projected/b1122c44-90ba-41b2-a10a-72e917ae092e-kube-api-access-567d8\") pod \"payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26\" (UID: \"b1122c44-90ba-41b2-a10a-72e917ae092e\") " pod="openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26" Nov 02 02:04:50.798453 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.798421 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26" Nov 02 02:04:50.909333 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:50.909299 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42" Nov 02 02:04:51.012461 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:51.012391 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:51.177634 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:51.177600 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26"] Nov 02 02:04:51.297255 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:51.297219 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42"] Nov 02 02:04:51.298486 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:04:51.298459 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85abf3dc_3eea_41cc_bbaf_2d0208adc731.slice/crio-38d27b05848dfff5d644734672181b62845298fb892e9de948f5a2ba29febbd1 WatchSource:0}: Error finding container 38d27b05848dfff5d644734672181b62845298fb892e9de948f5a2ba29febbd1: Status 404 returned error can't find the container with id 38d27b05848dfff5d644734672181b62845298fb892e9de948f5a2ba29febbd1 Nov 02 02:04:51.452791 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:51.452764 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f20c0468-c6b6-4718-b3f2-718c8c044fb9" path="/var/lib/kubelet/pods/f20c0468-c6b6-4718-b3f2-718c8c044fb9/volumes" Nov 02 02:04:51.461738 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:51.461705 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26" event={"ID":"b1122c44-90ba-41b2-a10a-72e917ae092e","Type":"ContainerStarted","Data":"56dff402c110ad69e57c4e1ef97105f76a69fbeb052f8e40e1ac329a43e39714"} Nov 02 02:04:51.462177 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:51.461759 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26" event={"ID":"b1122c44-90ba-41b2-a10a-72e917ae092e","Type":"ContainerStarted","Data":"1b004f6675b7d81d857d212bc9fb99983b7542aff930213f864b3e360cbf077e"} Nov 02 02:04:51.462177 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:51.461892 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26" Nov 02 02:04:51.463441 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:51.463415 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42" event={"ID":"85abf3dc-3eea-41cc-bbaf-2d0208adc731","Type":"ContainerStarted","Data":"dbbd4c2cf96fc8ce0fdce552c0f0b4d62c4f4df932a577fd716143281d560d14"} Nov 02 02:04:51.463592 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:51.463475 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42" Nov 02 02:04:51.463592 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:51.463499 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42" event={"ID":"85abf3dc-3eea-41cc-bbaf-2d0208adc731","Type":"ContainerStarted","Data":"38d27b05848dfff5d644734672181b62845298fb892e9de948f5a2ba29febbd1"} Nov 02 02:04:51.499563 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:51.499538 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:51.583653 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:51.583543 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26" podStartSLOduration=1.583528801 podStartE2EDuration="1.583528801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:04:51.581794313 +0000 UTC m=+2408.253046995" watchObservedRunningTime="2026-11-02 02:04:51.583528801 +0000 UTC m=+2408.254781508" Nov 02 02:04:51.644271 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:51.644243 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:51.709923 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:51.709858 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-07ac70d0-6fc99ffb9b-lrlj5" podUID="f20c0468-c6b6-4718-b3f2-718c8c044fb9" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.126:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 02:04:51.715095 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:51.715031 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42" podStartSLOduration=1.715011579 podStartE2EDuration="1.715011579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:04:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:04:51.701084905 +0000 UTC m=+2408.372337586" watchObservedRunningTime="2026-11-02 02:04:51.715011579 +0000 UTC m=+2408.386264261" Nov 02 02:04:52.369108 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:52.369078 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq" Nov 02 02:04:52.407415 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:52.407387 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" Nov 02 02:04:54.173523 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:04:54.173488 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:04:55.502517 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:55.502484 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:04:56.067750 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:04:56.067713 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:00.314037 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:00.314001 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:00.576119 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:00.576024 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:01.228153 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:01.228111 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:02.473021 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.472993 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42" Nov 02 02:05:02.473568 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.473393 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26" Nov 02 02:05:02.650604 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.650567 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4"] Nov 02 02:05:02.656939 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.656908 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.702990 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.702958 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:02.713792 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.713745 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4"] Nov 02 02:05:02.714933 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.714642 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-envoy\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.715110 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.715021 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-credential-socket\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.715110 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.714830 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c78625-openshift-default-dockercfg-wgkl5\"" Nov 02 02:05:02.715250 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.715178 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdc7t\" (UniqueName: \"kubernetes.io/projected/e16f31d3-d566-4620-a625-097e8ee9e28b-kube-api-access-bdc7t\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.715250 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.715228 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16f31d3-d566-4620-a625-097e8ee9e28b-istiod-ca-cert\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.715351 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.715266 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-data\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.715387 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.715348 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-workload-certs\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.715453 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.715429 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-podinfo\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.715510 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.715476 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-token\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.715570 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.715535 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-workload-socket\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.816948 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.816912 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-podinfo\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.816948 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.816954 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-token\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.817256 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.817004 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-workload-socket\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.817256 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.817056 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-envoy\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.817256 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.817105 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-credential-socket\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.817256 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.817207 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdc7t\" (UniqueName: \"kubernetes.io/projected/e16f31d3-d566-4620-a625-097e8ee9e28b-kube-api-access-bdc7t\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.817256 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.817251 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16f31d3-d566-4620-a625-097e8ee9e28b-istiod-ca-cert\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.817459 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.817285 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-data\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.817459 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.817316 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-workload-certs\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.817573 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.817552 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-workload-socket\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.817623 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.817607 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-credential-socket\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.817709 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.817645 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-workload-certs\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.817849 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.817824 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-data\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.818052 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.818033 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16f31d3-d566-4620-a625-097e8ee9e28b-istiod-ca-cert\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.819637 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.819609 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-envoy\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.819831 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.819812 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-podinfo\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.880219 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.880189 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-token\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.880372 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.880325 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdc7t\" (UniqueName: \"kubernetes.io/projected/e16f31d3-d566-4620-a625-097e8ee9e28b-kube-api-access-bdc7t\") pod \"e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:02.957255 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.957225 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:02.974295 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:02.974266 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:03.408768 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:03.408740 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4"] Nov 02 02:05:03.410229 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:05:03.410193 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16f31d3_d566_4620_a625_097e8ee9e28b.slice/crio-35c75f765cb763797606bab41c2657eec02c455a8855c18bd594788790f09eb9 WatchSource:0}: Error finding container 35c75f765cb763797606bab41c2657eec02c455a8855c18bd594788790f09eb9: Status 404 returned error can't find the container with id 35c75f765cb763797606bab41c2657eec02c455a8855c18bd594788790f09eb9 Nov 02 02:05:03.412912 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:03.412882 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:05:03.413027 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:03.412964 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:05:03.413027 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:03.413008 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:05:03.540648 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:03.540608 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" event={"ID":"e16f31d3-d566-4620-a625-097e8ee9e28b","Type":"ContainerStarted","Data":"a4428ddeda1e2cd7c8ca84a7f306996923b45e4d6282509ad6c62c4076c5bcd7"} Nov 02 02:05:03.541160 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:03.540665 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" event={"ID":"e16f31d3-d566-4620-a625-097e8ee9e28b","Type":"ContainerStarted","Data":"35c75f765cb763797606bab41c2657eec02c455a8855c18bd594788790f09eb9"} Nov 02 02:05:03.665206 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:03.665075 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" podStartSLOduration=1.665058749 podStartE2EDuration="1.665058749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:05:03.663402134 +0000 UTC m=+2420.334654850" watchObservedRunningTime="2026-11-02 02:05:03.665058749 +0000 UTC m=+2420.336311426" Nov 02 02:05:03.975101 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:03.975007 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:04.981485 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:04.981446 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:05.436146 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:05.436109 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:05.555428 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:05.555400 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:05.556568 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:05.556545 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:05:06.433661 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:06.433629 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:06.615243 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:06.615204 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:06.773083 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:06.773050 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:10.872470 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:10.872431 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:16.101861 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:16.101801 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:21.067343 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:21.067304 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:24.173931 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:05:24.173895 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:05:26.243427 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:26.243392 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:31.099340 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:31.099303 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:33.623946 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:33.623904 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:34.113380 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:34.113341 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:34.747949 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:34.747907 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:35.007393 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:35.007304 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:36.378287 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:36.378251 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:38.836633 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:38.836546 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:39.258162 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:39.258072 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:39.532950 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:39.532913 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:39.767505 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:39.767448 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:39.969768 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:39.969654 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:41.126657 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:41.126597 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:42.947314 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:42.947277 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:43.212089 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:43.212007 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:43.498529 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:43.498461 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:44.164540 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:44.164505 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:45.611846 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:45.611792 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:45.764784 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:45.764754 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:45.930586 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:45.930489 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:46.086052 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:46.086018 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:46.700164 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:46.700132 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:47.157630 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:47.157603 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:47.997963 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:47.997928 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:48.205084 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:48.205054 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:48.369880 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:48.369827 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:51.086190 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:51.086154 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:51.241134 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:51.241102 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:51.441808 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:51.441720 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv"] Nov 02 02:05:51.441993 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:51.441925 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" podUID="c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" containerName="istio-proxy" containerID="cri-o://e0e414d7fd49c210ddf30c115581a93bd5c5889d055ae5bd7bf0c447b01ee479" gracePeriod=30 Nov 02 02:05:51.458025 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:51.457990 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:52.611455 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:52.611418 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:52.793288 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:52.793205 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:52.960937 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:52.960852 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:53.412224 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:53.412185 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:54.173081 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:05:54.173045 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:05:55.704875 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:55.704833 5044 patch_prober.go:29] interesting pod/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 02:05:55.705280 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:55.704901 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" podUID="c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 02:05:56.703998 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.703974 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 02:05:56.784755 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.784725 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istiod-ca-cert\") pod \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " Nov 02 02:05:56.785158 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.784767 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-token\" (UniqueName: \"kubernetes.io/projected/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-token\") pod \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " Nov 02 02:05:56.785158 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.784809 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-envoy\") pod \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " Nov 02 02:05:56.785158 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.784922 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-credential-socket\") pod \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " Nov 02 02:05:56.785158 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.784971 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-podinfo\") pod \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " Nov 02 02:05:56.785158 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.785044 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-workload-socket\") pod \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " Nov 02 02:05:56.785158 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.785082 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-kube-api-access-l6zj8\" (UniqueName: \"kubernetes.io/projected/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-kube-api-access-l6zj8\") pod \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " Nov 02 02:05:56.785158 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.785084 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istiod-ca-cert" pod "c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" (UID: "c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:05:56.785158 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.785117 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-data\") pod \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " Nov 02 02:05:56.785604 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.785166 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-workload-certs\") pod \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\" (UID: \"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa\") " Nov 02 02:05:56.785604 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.785270 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-credential-socket" pod "c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" (UID: "c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:05:56.785604 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.785404 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-workload-socket" pod "c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" (UID: "c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:05:56.785604 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.785503 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-data" pod "c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" (UID: "c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:05:56.785851 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.785619 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-workload-certs" pod "c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" (UID: "c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:05:56.785851 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.785762 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:05:56.785851 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.785786 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:05:56.785851 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.785801 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:05:56.785851 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.785816 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:05:56.785851 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.785829 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:05:56.787379 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.787347 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-token" pod "c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" (UID: "c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:05:56.787379 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.787357 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-podinfo" pod "c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" (UID: "c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 02:05:56.787653 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.787627 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-kube-api-access-l6zj8" pod "c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" (UID: "c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa"). InnerVolumeSpecName "kube-api-access-l6zj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:05:56.787769 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.787737 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-envoy" pod "c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" (UID: "c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:05:56.886874 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.886843 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:05:56.886874 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.886873 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:05:56.887062 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.886886 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6zj8\" (UniqueName: \"kubernetes.io/projected/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-kube-api-access-l6zj8\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:05:56.887062 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.886899 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:05:56.898390 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.898359 5044 generic.go:356] "Generic (PLEG): container finished" podID="c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" containerID="e0e414d7fd49c210ddf30c115581a93bd5c5889d055ae5bd7bf0c447b01ee479" exitCode=0 Nov 02 02:05:56.898518 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.898428 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" Nov 02 02:05:56.898518 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.898439 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" event={"ID":"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa","Type":"ContainerDied","Data":"e0e414d7fd49c210ddf30c115581a93bd5c5889d055ae5bd7bf0c447b01ee479"} Nov 02 02:05:56.898518 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.898491 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv" event={"ID":"c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa","Type":"ContainerDied","Data":"a4ec4635ef576fd67611c9858b531e316ee30072ab07b1b6c50e91441b5adfa3"} Nov 02 02:05:56.898518 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.898514 5044 scope.go:122] "RemoveContainer" containerID="e0e414d7fd49c210ddf30c115581a93bd5c5889d055ae5bd7bf0c447b01ee479" Nov 02 02:05:56.914907 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.914888 5044 scope.go:122] "RemoveContainer" containerID="cd4da6ec8f1b51383423c155fb319292fa6c7bf32aa9b04bf546d2bccb7655b7" Nov 02 02:05:56.924033 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.924009 5044 scope.go:122] "RemoveContainer" containerID="e0e414d7fd49c210ddf30c115581a93bd5c5889d055ae5bd7bf0c447b01ee479" Nov 02 02:05:56.924362 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:05:56.924340 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e414d7fd49c210ddf30c115581a93bd5c5889d055ae5bd7bf0c447b01ee479\": container with ID starting with e0e414d7fd49c210ddf30c115581a93bd5c5889d055ae5bd7bf0c447b01ee479 not found: ID does not exist" containerID="e0e414d7fd49c210ddf30c115581a93bd5c5889d055ae5bd7bf0c447b01ee479" Nov 02 02:05:56.924448 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.924369 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e414d7fd49c210ddf30c115581a93bd5c5889d055ae5bd7bf0c447b01ee479"} err="failed to get container status \"e0e414d7fd49c210ddf30c115581a93bd5c5889d055ae5bd7bf0c447b01ee479\": rpc error: code = NotFound desc = could not find container \"e0e414d7fd49c210ddf30c115581a93bd5c5889d055ae5bd7bf0c447b01ee479\": container with ID starting with e0e414d7fd49c210ddf30c115581a93bd5c5889d055ae5bd7bf0c447b01ee479 not found: ID does not exist" Nov 02 02:05:56.924448 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.924388 5044 scope.go:122] "RemoveContainer" containerID="cd4da6ec8f1b51383423c155fb319292fa6c7bf32aa9b04bf546d2bccb7655b7" Nov 02 02:05:56.924706 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:05:56.924657 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4da6ec8f1b51383423c155fb319292fa6c7bf32aa9b04bf546d2bccb7655b7\": container with ID starting with cd4da6ec8f1b51383423c155fb319292fa6c7bf32aa9b04bf546d2bccb7655b7 not found: ID does not exist" containerID="cd4da6ec8f1b51383423c155fb319292fa6c7bf32aa9b04bf546d2bccb7655b7" Nov 02 02:05:56.924804 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:56.924710 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4da6ec8f1b51383423c155fb319292fa6c7bf32aa9b04bf546d2bccb7655b7"} err="failed to get container status \"cd4da6ec8f1b51383423c155fb319292fa6c7bf32aa9b04bf546d2bccb7655b7\": rpc error: code = NotFound desc = could not find container \"cd4da6ec8f1b51383423c155fb319292fa6c7bf32aa9b04bf546d2bccb7655b7\": container with ID starting with cd4da6ec8f1b51383423c155fb319292fa6c7bf32aa9b04bf546d2bccb7655b7 not found: ID does not exist" Nov 02 02:05:57.069224 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:57.069140 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv"] Nov 02 02:05:57.072116 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:57.072088 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f276c3-openshift-default-698cfdddf-vp2wv"] Nov 02 02:05:57.452680 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:57.452594 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" path="/var/lib/kubelet/pods/c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa/volumes" Nov 02 02:05:58.338576 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:58.338543 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:58.499481 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:58.499448 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:58.649082 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:58.649009 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:05:59.923239 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:05:59.923204 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:00.079405 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:00.079379 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:03.266749 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:03.266714 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:04.031489 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:04.031452 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:04.219339 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:04.219305 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:04.448421 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:04.448343 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt"] Nov 02 02:06:04.448970 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:04.448955 5044 cpu_manager.go:414] "containerMap: removing container" podUID="c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" containerName="istio-proxy" Nov 02 02:06:04.448970 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:04.448970 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" containerName="istio-proxy" Nov 02 02:06:04.449107 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:04.449095 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" containerName="istio-proxy" Nov 02 02:06:04.449143 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:04.449110 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" containerName="istio-proxy" Nov 02 02:06:04.453867 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:04.453842 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt" Nov 02 02:06:04.512023 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:04.511989 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt"] Nov 02 02:06:04.558872 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:04.558838 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c78625-dockercfg-28cgt\"" Nov 02 02:06:04.559277 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:04.559210 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c78625\"" Nov 02 02:06:04.585950 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:04.585914 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj5z8\" (UniqueName: \"kubernetes.io/projected/2e26e23b-5900-423c-9e14-7122ce904c81-kube-api-access-zj5z8\") pod \"payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt\" (UID: \"2e26e23b-5900-423c-9e14-7122ce904c81\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt" Nov 02 02:06:04.586848 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:04.586816 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e26e23b-5900-423c-9e14-7122ce904c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt\" (UID: \"2e26e23b-5900-423c-9e14-7122ce904c81\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt" Nov 02 02:06:04.688983 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:04.688942 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj5z8\" (UniqueName: \"kubernetes.io/projected/2e26e23b-5900-423c-9e14-7122ce904c81-kube-api-access-zj5z8\") pod \"payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt\" (UID: \"2e26e23b-5900-423c-9e14-7122ce904c81\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt" Nov 02 02:06:04.689182 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:04.689012 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e26e23b-5900-423c-9e14-7122ce904c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt\" (UID: \"2e26e23b-5900-423c-9e14-7122ce904c81\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt" Nov 02 02:06:04.689894 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:04.689866 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e26e23b-5900-423c-9e14-7122ce904c81-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt\" (UID: \"2e26e23b-5900-423c-9e14-7122ce904c81\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt" Nov 02 02:06:04.751307 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:04.751233 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj5z8\" (UniqueName: \"kubernetes.io/projected/2e26e23b-5900-423c-9e14-7122ce904c81-kube-api-access-zj5z8\") pod \"payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt\" (UID: \"2e26e23b-5900-423c-9e14-7122ce904c81\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt" Nov 02 02:06:04.768821 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:04.768791 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt" Nov 02 02:06:05.152959 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:05.152931 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt"] Nov 02 02:06:05.154445 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:06:05.154415 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e26e23b_5900_423c_9e14_7122ce904c81.slice/crio-1b52f46838f2ad9b1186ec8d3c79bfe93c07ce0fb5886bd0b1c7c60813b05e50 WatchSource:0}: Error finding container 1b52f46838f2ad9b1186ec8d3c79bfe93c07ce0fb5886bd0b1c7c60813b05e50: Status 404 returned error can't find the container with id 1b52f46838f2ad9b1186ec8d3c79bfe93c07ce0fb5886bd0b1c7c60813b05e50 Nov 02 02:06:05.954433 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:05.954396 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt" event={"ID":"2e26e23b-5900-423c-9e14-7122ce904c81","Type":"ContainerStarted","Data":"79df2b34d3ea4ca0a623369c1454d510cf86b19041dfadddfb8265fed3593b94"} Nov 02 02:06:05.954834 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:05.954450 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt" event={"ID":"2e26e23b-5900-423c-9e14-7122ce904c81","Type":"ContainerStarted","Data":"1b52f46838f2ad9b1186ec8d3c79bfe93c07ce0fb5886bd0b1c7c60813b05e50"} Nov 02 02:06:05.954834 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:05.954493 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt" Nov 02 02:06:06.076280 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:06.076227 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt" podStartSLOduration=2.076212911 podStartE2EDuration="2.076212911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:06:06.074163853 +0000 UTC m=+2482.745416535" watchObservedRunningTime="2026-11-02 02:06:06.076212911 +0000 UTC m=+2482.747465592" Nov 02 02:06:08.188769 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:08.188737 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:12.996733 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:12.996697 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:16.963518 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:16.963491 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt" Nov 02 02:06:17.772450 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:17.772414 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:22.603095 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:22.603040 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:24.172204 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:06:24.172170 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:06:27.395488 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:27.395450 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:32.167901 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:32.167871 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:35.177032 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.176996 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj"] Nov 02 02:06:35.180071 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.177794 5044 cpu_manager.go:414] "containerMap: removing container" podUID="c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" containerName="istio-proxy" Nov 02 02:06:35.180071 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.177810 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1af3dd7-0ab8-4feb-a39c-7e15f9d2c7fa" containerName="istio-proxy" Nov 02 02:06:35.181462 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.181447 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.261504 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.261466 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.261504 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.261510 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.261800 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.261537 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.261800 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.261738 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rxfv\" (UniqueName: \"kubernetes.io/projected/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-kube-api-access-5rxfv\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.261923 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.261837 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.261923 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.261884 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.261923 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.261918 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.262099 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.262063 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.262160 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.262103 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.296361 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.296321 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj"] Nov 02 02:06:35.363683 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.363629 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rxfv\" (UniqueName: \"kubernetes.io/projected/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-kube-api-access-5rxfv\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.363877 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.363805 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.363877 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.363856 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.364017 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.363890 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.364017 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.363939 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.364017 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.363965 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.364190 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.364065 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.364190 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.364103 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.364190 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.364145 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.364477 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.364454 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.364569 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.364499 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.364788 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.364769 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.365341 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.365318 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.365558 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.365539 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.367775 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.367753 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.367870 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.367815 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.426755 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.426726 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.427546 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.427489 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rxfv\" (UniqueName: \"kubernetes.io/projected/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-kube-api-access-5rxfv\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9fjhj\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.496481 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.496456 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:35.929380 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.929355 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj"] Nov 02 02:06:35.930201 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:06:35.930170 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ee089e_cf8a_4d98_b6b5_4e1f374c3cbc.slice/crio-983cbf2b6a1689b34c47602941870d8104b449d5b5c2b767347938a1e474042b WatchSource:0}: Error finding container 983cbf2b6a1689b34c47602941870d8104b449d5b5c2b767347938a1e474042b: Status 404 returned error can't find the container with id 983cbf2b6a1689b34c47602941870d8104b449d5b5c2b767347938a1e474042b Nov 02 02:06:35.932974 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.932932 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:06:35.933104 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.933008 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:06:35.933104 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:35.933054 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:06:36.196790 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.196649 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" event={"ID":"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc","Type":"ContainerStarted","Data":"50b67d2dd2612e3b9b0c3f71d45475217ef8439fe8266bb5f4645e15cd2b6df2"} Nov 02 02:06:36.196790 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.196754 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" event={"ID":"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc","Type":"ContainerStarted","Data":"983cbf2b6a1689b34c47602941870d8104b449d5b5c2b767347938a1e474042b"} Nov 02 02:06:36.323404 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.323352 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" podStartSLOduration=1.3233379840000001 podStartE2EDuration="1.323337984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:06:36.320838813 +0000 UTC m=+2512.992091496" watchObservedRunningTime="2026-11-02 02:06:36.323337984 +0000 UTC m=+2512.994590666" Nov 02 02:06:36.497570 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.497484 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:36.499611 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.499581 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 02:06:36.499781 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.499645 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" podUID="e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 02:06:36.643274 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.643235 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92"] Nov 02 02:06:36.647168 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.647144 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" Nov 02 02:06:36.686230 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.686203 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380ec44f-2ae2-4c12-9f95-5c6e424de57c-maas-api-tls\") pod \"maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92\" (UID: \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" Nov 02 02:06:36.686407 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.686235 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gctx8\" (UniqueName: \"kubernetes.io/projected/380ec44f-2ae2-4c12-9f95-5c6e424de57c-kube-api-access-gctx8\") pod \"maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92\" (UID: \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" Nov 02 02:06:36.686475 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.686405 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/380ec44f-2ae2-4c12-9f95-5c6e424de57c-metrics-certs\") pod \"maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92\" (UID: \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" Nov 02 02:06:36.686540 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.686472 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380ec44f-2ae2-4c12-9f95-5c6e424de57c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92\" (UID: \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" Nov 02 02:06:36.702919 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.702895 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-361fcc-serving-cert\"" Nov 02 02:06:36.707138 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.707016 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92"] Nov 02 02:06:36.796528 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.796489 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/380ec44f-2ae2-4c12-9f95-5c6e424de57c-metrics-certs\") pod \"maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92\" (UID: \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" Nov 02 02:06:36.796760 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.796569 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380ec44f-2ae2-4c12-9f95-5c6e424de57c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92\" (UID: \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" Nov 02 02:06:36.796760 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.796663 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380ec44f-2ae2-4c12-9f95-5c6e424de57c-maas-api-tls\") pod \"maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92\" (UID: \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" Nov 02 02:06:36.796760 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.796703 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gctx8\" (UniqueName: \"kubernetes.io/projected/380ec44f-2ae2-4c12-9f95-5c6e424de57c-kube-api-access-gctx8\") pod \"maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92\" (UID: \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" Nov 02 02:06:36.799526 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.799493 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380ec44f-2ae2-4c12-9f95-5c6e424de57c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92\" (UID: \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" Nov 02 02:06:36.799651 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:06:36.799612 5044 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-serving-cert: secret "maas-api-e2e-inf-b-361fcc-serving-cert" not found Nov 02 02:06:36.815745 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:06:36.815714 5044 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/380ec44f-2ae2-4c12-9f95-5c6e424de57c-maas-api-tls podName:380ec44f-2ae2-4c12-9f95-5c6e424de57c nodeName:}" failed. No retries permitted until 2026-11-02 02:06:37.299652745 +0000 UTC m=+2513.970905419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/380ec44f-2ae2-4c12-9f95-5c6e424de57c-maas-api-tls") pod "maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" (UID: "380ec44f-2ae2-4c12-9f95-5c6e424de57c") : secret "maas-api-e2e-inf-b-361fcc-serving-cert" not found Nov 02 02:06:36.817437 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.817409 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/380ec44f-2ae2-4c12-9f95-5c6e424de57c-metrics-certs\") pod \"maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92\" (UID: \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" Nov 02 02:06:36.847125 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.847093 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:36.867843 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:36.867741 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gctx8\" (UniqueName: \"kubernetes.io/projected/380ec44f-2ae2-4c12-9f95-5c6e424de57c-kube-api-access-gctx8\") pod \"maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92\" (UID: \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" Nov 02 02:06:37.005757 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:37.005721 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:37.155934 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:37.155847 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:37.305095 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:37.305048 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380ec44f-2ae2-4c12-9f95-5c6e424de57c-maas-api-tls\") pod \"maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92\" (UID: \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" Nov 02 02:06:37.307863 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:37.307834 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:37.308032 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:37.307891 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380ec44f-2ae2-4c12-9f95-5c6e424de57c-maas-api-tls\") pod \"maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92\" (UID: \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" Nov 02 02:06:37.498180 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:37.498086 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 02:06:37.498180 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:37.498143 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" podUID="e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 02:06:37.567695 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:37.565858 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" Nov 02 02:06:38.014121 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:38.014089 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92"] Nov 02 02:06:38.214298 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:38.214262 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" event={"ID":"380ec44f-2ae2-4c12-9f95-5c6e424de57c","Type":"ContainerStarted","Data":"f1aeca22a67a65e3aa80c497e12c21835e51d6abb5caa3cc0f2f3c8d2b350e68"} Nov 02 02:06:38.497843 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:38.497806 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 02:06:38.498268 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:38.497865 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" podUID="e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 02:06:39.224465 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:39.224349 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" event={"ID":"380ec44f-2ae2-4c12-9f95-5c6e424de57c","Type":"ContainerStarted","Data":"2bbb20a2873a92c54011d6158991dbc24b77643c63f05c1cee82326e097f3b1e"} Nov 02 02:06:39.224465 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:39.224434 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" Nov 02 02:06:39.358005 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:39.357946 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" podStartSLOduration=2.431142109 podStartE2EDuration="3.357932783s" totalImagesPullingTime="926.790674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:06:39.357263657 +0000 UTC m=+2516.028516338" watchObservedRunningTime="2026-11-02 02:06:39.357932783 +0000 UTC m=+2516.029185464" Nov 02 02:06:39.514772 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:39.514722 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 02:06:39.516707 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:39.516657 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" podUID="e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 02:06:40.497019 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:40.496973 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 02:06:40.497234 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:40.497034 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" podUID="e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 02:06:41.497610 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:41.497572 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 02:06:41.498049 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:41.497635 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" podUID="e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 02:06:42.230507 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:42.230473 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:42.498216 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:42.498136 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 02:06:42.498216 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:42.498191 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" podUID="e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 02:06:42.755931 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:42.755849 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:42.937007 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:42.936974 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:43.178497 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:43.178460 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5"] Nov 02 02:06:43.183011 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:43.182988 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5" Nov 02 02:06:43.240681 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:43.240634 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5"] Nov 02 02:06:43.281343 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:43.281310 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v92vm\" (UniqueName: \"kubernetes.io/projected/4e55a287-c53e-49bf-88ad-21e4ede955a6-kube-api-access-v92vm\") pod \"payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5\" (UID: \"4e55a287-c53e-49bf-88ad-21e4ede955a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5" Nov 02 02:06:43.281513 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:43.281354 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e55a287-c53e-49bf-88ad-21e4ede955a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5\" (UID: \"4e55a287-c53e-49bf-88ad-21e4ede955a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5" Nov 02 02:06:43.288223 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:43.288201 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-361fcc\"" Nov 02 02:06:43.382514 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:43.382475 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v92vm\" (UniqueName: \"kubernetes.io/projected/4e55a287-c53e-49bf-88ad-21e4ede955a6-kube-api-access-v92vm\") pod \"payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5\" (UID: \"4e55a287-c53e-49bf-88ad-21e4ede955a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5" Nov 02 02:06:43.382727 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:43.382528 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e55a287-c53e-49bf-88ad-21e4ede955a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5\" (UID: \"4e55a287-c53e-49bf-88ad-21e4ede955a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5" Nov 02 02:06:43.383321 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:43.383300 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e55a287-c53e-49bf-88ad-21e4ede955a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5\" (UID: \"4e55a287-c53e-49bf-88ad-21e4ede955a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5" Nov 02 02:06:43.446802 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:43.446724 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v92vm\" (UniqueName: \"kubernetes.io/projected/4e55a287-c53e-49bf-88ad-21e4ede955a6-kube-api-access-v92vm\") pod \"payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5\" (UID: \"4e55a287-c53e-49bf-88ad-21e4ede955a6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5" Nov 02 02:06:43.497339 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:43.497300 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 02 02:06:43.497495 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:43.497364 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" podUID="e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 02 02:06:43.499723 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:43.499700 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5" Nov 02 02:06:43.883790 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:43.883761 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5"] Nov 02 02:06:43.885644 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:06:43.885615 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e55a287_c53e_49bf_88ad_21e4ede955a6.slice/crio-0fd6a497facac5125c0c01317f7af41cc223353cf9b35f818a219f99d1024bd2 WatchSource:0}: Error finding container 0fd6a497facac5125c0c01317f7af41cc223353cf9b35f818a219f99d1024bd2: Status 404 returned error can't find the container with id 0fd6a497facac5125c0c01317f7af41cc223353cf9b35f818a219f99d1024bd2 Nov 02 02:06:44.267253 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:44.266085 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5" event={"ID":"4e55a287-c53e-49bf-88ad-21e4ede955a6","Type":"ContainerStarted","Data":"5af640670d569aa7c6499c939c8ad4a03923a7409ce127979df43993eb423eac"} Nov 02 02:06:44.267253 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:44.266155 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5" event={"ID":"4e55a287-c53e-49bf-88ad-21e4ede955a6","Type":"ContainerStarted","Data":"0fd6a497facac5125c0c01317f7af41cc223353cf9b35f818a219f99d1024bd2"} Nov 02 02:06:44.267253 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:44.267188 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5" Nov 02 02:06:44.396210 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:44.396146 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5" podStartSLOduration=1.396125846 podStartE2EDuration="1.396125846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:06:44.395350598 +0000 UTC m=+2521.066603283" watchObservedRunningTime="2026-11-02 02:06:44.396125846 +0000 UTC m=+2521.067378529" Nov 02 02:06:45.131272 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:45.131238 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:45.132414 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:45.132385 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:45.133084 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:45.133063 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:06:45.237796 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:45.237769 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" Nov 02 02:06:47.178718 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:47.178457 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:47.527224 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:47.527194 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:51.597832 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:51.597793 5044 patch_prober.go:29] interesting pod/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:06:51.598224 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:51.597850 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" podUID="e16f31d3-d566-4620-a625-097e8ee9e28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:06:51.830213 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:51.830159 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:52.043803 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:52.043763 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:52.191259 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:52.191219 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:53.354128 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:53.354051 5044 generic.go:356] "Generic (PLEG): container finished" podID="e16f31d3-d566-4620-a625-097e8ee9e28b" containerID="a4428ddeda1e2cd7c8ca84a7f306996923b45e4d6282509ad6c62c4076c5bcd7" exitCode=0 Nov 02 02:06:53.354545 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:53.354129 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" event={"ID":"e16f31d3-d566-4620-a625-097e8ee9e28b","Type":"ContainerDied","Data":"a4428ddeda1e2cd7c8ca84a7f306996923b45e4d6282509ad6c62c4076c5bcd7"} Nov 02 02:06:53.354597 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:53.354565 5044 scope.go:122] "RemoveContainer" containerID="a4428ddeda1e2cd7c8ca84a7f306996923b45e4d6282509ad6c62c4076c5bcd7" Nov 02 02:06:53.356036 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:53.356005 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:06:53.356114 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:53.356078 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:06:53.356152 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:53.356110 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:06:54.174391 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:06:54.174353 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:06:54.370691 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:54.369256 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" event={"ID":"e16f31d3-d566-4620-a625-097e8ee9e28b","Type":"ContainerStarted","Data":"b21d5788b748885131e2d7156bf65dde524f023a4d6fe7e26df32203e4cdff56"} Nov 02 02:06:54.979694 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:54.979096 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:06:54.979694 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:54.979226 5044 patch_prober.go:29] interesting pod/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 02 02:06:54.979694 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:54.979272 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" podUID="e16f31d3-d566-4620-a625-097e8ee9e28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 02 02:06:55.429455 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:55.429426 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:55.591721 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:55.586778 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:55.979070 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:55.979033 5044 patch_prober.go:29] interesting pod/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 02 02:06:55.979381 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:55.979355 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" podUID="e16f31d3-d566-4620-a625-097e8ee9e28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 02 02:06:56.308176 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:56.308148 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5" Nov 02 02:06:56.760812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:56.760780 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:56.975700 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:56.975014 5044 patch_prober.go:29] interesting pod/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 02 02:06:56.975700 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:56.975088 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" podUID="e16f31d3-d566-4620-a625-097e8ee9e28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 02 02:06:57.445811 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:57.445778 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:57.650695 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:57.648879 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:06:58.007461 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:58.006981 5044 patch_prober.go:29] interesting pod/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 02 02:06:58.007461 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:58.007058 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" podUID="e16f31d3-d566-4620-a625-097e8ee9e28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 02 02:06:58.075697 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:58.053372 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9"] Nov 02 02:06:58.099829 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:58.099790 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9" Nov 02 02:06:58.110658 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:58.110625 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9"] Nov 02 02:06:58.155234 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:58.155203 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9039bf\"" Nov 02 02:06:58.194740 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:58.193329 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9039bf-dockercfg-2spbx\"" Nov 02 02:06:58.216595 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:58.216530 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bt8d\" (UniqueName: \"kubernetes.io/projected/581a88ec-7e87-4e19-b6d8-093d7f2b2cb7-kube-api-access-5bt8d\") pod \"payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9\" (UID: \"581a88ec-7e87-4e19-b6d8-093d7f2b2cb7\") " pod="openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9" Nov 02 02:06:58.216799 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:58.216624 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/581a88ec-7e87-4e19-b6d8-093d7f2b2cb7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9\" (UID: \"581a88ec-7e87-4e19-b6d8-093d7f2b2cb7\") " pod="openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9" Nov 02 02:06:58.318421 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:58.318380 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bt8d\" (UniqueName: \"kubernetes.io/projected/581a88ec-7e87-4e19-b6d8-093d7f2b2cb7-kube-api-access-5bt8d\") pod \"payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9\" (UID: \"581a88ec-7e87-4e19-b6d8-093d7f2b2cb7\") " pod="openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9" Nov 02 02:06:58.318608 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:58.318431 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/581a88ec-7e87-4e19-b6d8-093d7f2b2cb7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9\" (UID: \"581a88ec-7e87-4e19-b6d8-093d7f2b2cb7\") " pod="openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9" Nov 02 02:06:58.319118 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:58.319098 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/581a88ec-7e87-4e19-b6d8-093d7f2b2cb7-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9\" (UID: \"581a88ec-7e87-4e19-b6d8-093d7f2b2cb7\") " pod="openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9" Nov 02 02:06:58.387693 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:58.387638 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bt8d\" (UniqueName: \"kubernetes.io/projected/581a88ec-7e87-4e19-b6d8-093d7f2b2cb7-kube-api-access-5bt8d\") pod \"payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9\" (UID: \"581a88ec-7e87-4e19-b6d8-093d7f2b2cb7\") " pod="openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9" Nov 02 02:06:58.438171 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:58.438142 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9" Nov 02 02:06:58.846433 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:58.846404 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9"] Nov 02 02:06:58.848155 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:06:58.848128 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod581a88ec_7e87_4e19_b6d8_093d7f2b2cb7.slice/crio-973bde58a2efe3a1692ef206c2120c114ce2848a64afe3a2898d351973bf3f8d WatchSource:0}: Error finding container 973bde58a2efe3a1692ef206c2120c114ce2848a64afe3a2898d351973bf3f8d: Status 404 returned error can't find the container with id 973bde58a2efe3a1692ef206c2120c114ce2848a64afe3a2898d351973bf3f8d Nov 02 02:06:58.977696 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:58.975083 5044 patch_prober.go:29] interesting pod/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 02 02:06:58.977696 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:58.975148 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" podUID="e16f31d3-d566-4620-a625-097e8ee9e28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 02 02:06:59.425685 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:59.425629 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9" event={"ID":"581a88ec-7e87-4e19-b6d8-093d7f2b2cb7","Type":"ContainerStarted","Data":"11eefe167b370108aeb1b94c71d1ff6d11f73537a78be68f025a690b47185e24"} Nov 02 02:06:59.426136 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:59.425716 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9" event={"ID":"581a88ec-7e87-4e19-b6d8-093d7f2b2cb7","Type":"ContainerStarted","Data":"973bde58a2efe3a1692ef206c2120c114ce2848a64afe3a2898d351973bf3f8d"} Nov 02 02:06:59.426136 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:59.425750 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9" Nov 02 02:06:59.556224 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:59.555663 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9" podStartSLOduration=2.555644051 podStartE2EDuration="2.555644051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:06:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:06:59.550288844 +0000 UTC m=+2536.221541526" watchObservedRunningTime="2026-11-02 02:06:59.555644051 +0000 UTC m=+2536.226896734" Nov 02 02:06:59.978695 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:59.978652 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:06:59.979038 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:59.979019 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:06:59.979654 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:06:59.979633 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:07:01.657945 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:01.657910 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:05.468246 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:05.468214 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:06.683598 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:06.683550 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:09.224750 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:09.224712 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:10.423136 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:10.423101 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6"] Nov 02 02:07:10.435332 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:10.435306 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9" Nov 02 02:07:10.550728 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:10.550698 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:11.966813 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:11.966780 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:17.241693 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:17.241614 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:17.759538 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:17.759501 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9"] Nov 02 02:07:18.061747 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:18.048639 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:18.286316 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:18.286270 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:22.184211 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:22.184164 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:22.335262 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:22.335216 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:22.495813 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:22.495658 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:22.760926 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:22.760866 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9" podUID="bd419016-7531-4bcc-9f84-e07ee7a5d747" containerName="payload-pre-processing" containerID="cri-o://2f5e639657b2985439ad95cb4395ac336e743645325739fb6f58c0768b408865" gracePeriod=25 Nov 02 02:07:24.154116 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:24.154079 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9" podUID="bd419016-7531-4bcc-9f84-e07ee7a5d747" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 02:07:24.173140 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:07:24.173102 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:07:25.446036 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.445991 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" containerID="cri-o://45d7a0f24c32cdc4bb1cbcc70f79706c3179a62945f3762b883381797064e4fd" gracePeriod=45 Nov 02 02:07:25.679839 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.679806 5044 generic.go:356] "Generic (PLEG): container finished" podID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerID="45d7a0f24c32cdc4bb1cbcc70f79706c3179a62945f3762b883381797064e4fd" exitCode=0 Nov 02 02:07:25.680019 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.679881 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" event={"ID":"39920d19-93e7-46ad-a3a4-90a51b766ab3","Type":"ContainerDied","Data":"45d7a0f24c32cdc4bb1cbcc70f79706c3179a62945f3762b883381797064e4fd"} Nov 02 02:07:25.710713 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.710628 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 02:07:25.823778 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.823750 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-dshm\") pod \"39920d19-93e7-46ad-a3a4-90a51b766ab3\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " Nov 02 02:07:25.824026 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.823795 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39920d19-93e7-46ad-a3a4-90a51b766ab3-kube-api-access-n7vv9\" (UniqueName: \"kubernetes.io/projected/39920d19-93e7-46ad-a3a4-90a51b766ab3-kube-api-access-n7vv9\") pod \"39920d19-93e7-46ad-a3a4-90a51b766ab3\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " Nov 02 02:07:25.824026 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.823829 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39920d19-93e7-46ad-a3a4-90a51b766ab3-tls-certs\" (UniqueName: \"kubernetes.io/secret/39920d19-93e7-46ad-a3a4-90a51b766ab3-tls-certs\") pod \"39920d19-93e7-46ad-a3a4-90a51b766ab3\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " Nov 02 02:07:25.824026 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.823863 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-model-cache\") pod \"39920d19-93e7-46ad-a3a4-90a51b766ab3\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " Nov 02 02:07:25.824026 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.823933 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-tmp-dir\") pod \"39920d19-93e7-46ad-a3a4-90a51b766ab3\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " Nov 02 02:07:25.824252 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.824062 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-home\" (UniqueName: \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-home\") pod \"39920d19-93e7-46ad-a3a4-90a51b766ab3\" (UID: \"39920d19-93e7-46ad-a3a4-90a51b766ab3\") " Nov 02 02:07:25.824320 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.824236 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-model-cache" pod "39920d19-93e7-46ad-a3a4-90a51b766ab3" (UID: "39920d19-93e7-46ad-a3a4-90a51b766ab3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:07:25.824320 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.824290 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-tmp-dir" pod "39920d19-93e7-46ad-a3a4-90a51b766ab3" (UID: "39920d19-93e7-46ad-a3a4-90a51b766ab3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:07:25.824430 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.824399 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-home" pod "39920d19-93e7-46ad-a3a4-90a51b766ab3" (UID: "39920d19-93e7-46ad-a3a4-90a51b766ab3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:07:25.824616 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.824597 5044 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-model-cache\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:07:25.824688 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.824650 5044 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-tmp-dir\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:07:25.824824 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.824691 5044 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-home\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:07:25.826127 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.826108 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39920d19-93e7-46ad-a3a4-90a51b766ab3-tls-certs" pod "39920d19-93e7-46ad-a3a4-90a51b766ab3" (UID: "39920d19-93e7-46ad-a3a4-90a51b766ab3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:07:25.826195 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.826141 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39920d19-93e7-46ad-a3a4-90a51b766ab3-kube-api-access-n7vv9" pod "39920d19-93e7-46ad-a3a4-90a51b766ab3" (UID: "39920d19-93e7-46ad-a3a4-90a51b766ab3"). InnerVolumeSpecName "kube-api-access-n7vv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:07:25.826383 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.826362 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-dshm" pod "39920d19-93e7-46ad-a3a4-90a51b766ab3" (UID: "39920d19-93e7-46ad-a3a4-90a51b766ab3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:07:25.926464 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.926376 5044 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39920d19-93e7-46ad-a3a4-90a51b766ab3-dshm\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:07:25.926464 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.926418 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7vv9\" (UniqueName: \"kubernetes.io/projected/39920d19-93e7-46ad-a3a4-90a51b766ab3-kube-api-access-n7vv9\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:07:25.926464 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:25.926434 5044 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39920d19-93e7-46ad-a3a4-90a51b766ab3-tls-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:07:26.687389 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:26.687359 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" Nov 02 02:07:26.687849 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:26.687401 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6" event={"ID":"39920d19-93e7-46ad-a3a4-90a51b766ab3","Type":"ContainerDied","Data":"7f5f700fae5dc182578515bc8b94f9bdefe5b28c5834e01c75de9a60a4f9c4b8"} Nov 02 02:07:26.687849 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:26.687450 5044 scope.go:122] "RemoveContainer" containerID="45d7a0f24c32cdc4bb1cbcc70f79706c3179a62945f3762b883381797064e4fd" Nov 02 02:07:26.855902 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:26.855869 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6"] Nov 02 02:07:26.859558 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:26.859534 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-36a1e0/distinct-2-w2-36a1e0-kserve-84c864789d-jnpb6"] Nov 02 02:07:26.949149 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:26.949070 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:27.454683 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:27.454641 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" path="/var/lib/kubelet/pods/39920d19-93e7-46ad-a3a4-90a51b766ab3/volumes" Nov 02 02:07:31.921167 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:31.921131 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:34.153938 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:34.153901 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9" podUID="bd419016-7531-4bcc-9f84-e07ee7a5d747" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 02:07:36.788337 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:36.788291 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:41.640927 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:41.640893 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:43.036588 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:43.036564 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9" Nov 02 02:07:43.123966 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:43.123939 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd419016-7531-4bcc-9f84-e07ee7a5d747-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd419016-7531-4bcc-9f84-e07ee7a5d747-plugins-config-volume\") pod \"bd419016-7531-4bcc-9f84-e07ee7a5d747\" (UID: \"bd419016-7531-4bcc-9f84-e07ee7a5d747\") " Nov 02 02:07:43.124133 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:43.123982 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd419016-7531-4bcc-9f84-e07ee7a5d747-kube-api-access-7jsl9\" (UniqueName: \"kubernetes.io/projected/bd419016-7531-4bcc-9f84-e07ee7a5d747-kube-api-access-7jsl9\") pod \"bd419016-7531-4bcc-9f84-e07ee7a5d747\" (UID: \"bd419016-7531-4bcc-9f84-e07ee7a5d747\") " Nov 02 02:07:43.124333 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:43.124309 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd419016-7531-4bcc-9f84-e07ee7a5d747-plugins-config-volume" pod "bd419016-7531-4bcc-9f84-e07ee7a5d747" (UID: "bd419016-7531-4bcc-9f84-e07ee7a5d747"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:07:43.126079 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:43.126056 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd419016-7531-4bcc-9f84-e07ee7a5d747-kube-api-access-7jsl9" pod "bd419016-7531-4bcc-9f84-e07ee7a5d747" (UID: "bd419016-7531-4bcc-9f84-e07ee7a5d747"). InnerVolumeSpecName "kube-api-access-7jsl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:07:43.225265 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:43.225229 5044 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd419016-7531-4bcc-9f84-e07ee7a5d747-plugins-config-volume\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:07:43.225449 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:43.225267 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jsl9\" (UniqueName: \"kubernetes.io/projected/bd419016-7531-4bcc-9f84-e07ee7a5d747-kube-api-access-7jsl9\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:07:43.812266 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:43.812227 5044 generic.go:356] "Generic (PLEG): container finished" podID="bd419016-7531-4bcc-9f84-e07ee7a5d747" containerID="2f5e639657b2985439ad95cb4395ac336e743645325739fb6f58c0768b408865" exitCode=0 Nov 02 02:07:43.812458 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:43.812271 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9" Nov 02 02:07:43.812458 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:43.812316 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9" event={"ID":"bd419016-7531-4bcc-9f84-e07ee7a5d747","Type":"ContainerDied","Data":"2f5e639657b2985439ad95cb4395ac336e743645325739fb6f58c0768b408865"} Nov 02 02:07:43.812458 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:43.812366 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9" event={"ID":"bd419016-7531-4bcc-9f84-e07ee7a5d747","Type":"ContainerDied","Data":"1fea021420fcb9e41389fb012998912abf9049941e3f1017fa85bcfa85f88083"} Nov 02 02:07:43.812458 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:43.812390 5044 scope.go:122] "RemoveContainer" containerID="2f5e639657b2985439ad95cb4395ac336e743645325739fb6f58c0768b408865" Nov 02 02:07:43.821946 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:43.821930 5044 scope.go:122] "RemoveContainer" containerID="2f5e639657b2985439ad95cb4395ac336e743645325739fb6f58c0768b408865" Nov 02 02:07:43.822221 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:07:43.822201 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f5e639657b2985439ad95cb4395ac336e743645325739fb6f58c0768b408865\": container with ID starting with 2f5e639657b2985439ad95cb4395ac336e743645325739fb6f58c0768b408865 not found: ID does not exist" containerID="2f5e639657b2985439ad95cb4395ac336e743645325739fb6f58c0768b408865" Nov 02 02:07:43.822287 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:43.822233 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5e639657b2985439ad95cb4395ac336e743645325739fb6f58c0768b408865"} err="failed to get container status \"2f5e639657b2985439ad95cb4395ac336e743645325739fb6f58c0768b408865\": rpc error: code = NotFound desc = could not find container \"2f5e639657b2985439ad95cb4395ac336e743645325739fb6f58c0768b408865\": container with ID starting with 2f5e639657b2985439ad95cb4395ac336e743645325739fb6f58c0768b408865 not found: ID does not exist" Nov 02 02:07:43.981849 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:43.981813 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9"] Nov 02 02:07:44.033234 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:44.033201 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-36a1e0-7679b689d7-q8ld9"] Nov 02 02:07:45.453613 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:45.453577 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd419016-7531-4bcc-9f84-e07ee7a5d747" path="/var/lib/kubelet/pods/bd419016-7531-4bcc-9f84-e07ee7a5d747/volumes" Nov 02 02:07:46.968541 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:46.968507 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:47.288792 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:47.288718 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:47.624915 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:47.624828 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:47.773135 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:47.773099 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:47.976693 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:47.976606 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:48.462442 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:48.462404 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:50.632765 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:50.632718 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:50.793831 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:50.793800 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:51.928232 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:51.928195 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:54.027854 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:54.027811 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:54.171339 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:54.171309 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:54.173238 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:07:54.173210 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:07:55.327926 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:55.327891 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:07:56.912241 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:07:56.912177 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:02.120851 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:02.120815 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:03.834143 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:03.834110 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:03.977716 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:03.977664 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:07.237472 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:07.237436 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:08.413727 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:08.413688 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26"] Nov 02 02:08:08.523031 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:08.522999 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42"] Nov 02 02:08:08.774139 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:08.774106 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:08.916820 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:08.916786 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:09.929603 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:09.929549 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:11.995043 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:11.995010 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:13.414633 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:13.414572 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26" podUID="b1122c44-90ba-41b2-a10a-72e917ae092e" containerName="payload-processing" containerID="cri-o://56dff402c110ad69e57c4e1ef97105f76a69fbeb052f8e40e1ac329a43e39714" gracePeriod=25 Nov 02 02:08:13.523542 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:13.523506 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42" podUID="85abf3dc-3eea-41cc-bbaf-2d0208adc731" containerName="payload-pre-processing" containerID="cri-o://dbbd4c2cf96fc8ce0fdce552c0f0b4d62c4f4df932a577fd716143281d560d14" gracePeriod=25 Nov 02 02:08:13.679537 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:13.679462 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26" Nov 02 02:08:13.794632 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:13.794609 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42" Nov 02 02:08:13.802503 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:13.802477 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1122c44-90ba-41b2-a10a-72e917ae092e-kube-api-access-567d8\" (UniqueName: \"kubernetes.io/projected/b1122c44-90ba-41b2-a10a-72e917ae092e-kube-api-access-567d8\") pod \"b1122c44-90ba-41b2-a10a-72e917ae092e\" (UID: \"b1122c44-90ba-41b2-a10a-72e917ae092e\") " Nov 02 02:08:13.802658 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:13.802625 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1122c44-90ba-41b2-a10a-72e917ae092e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1122c44-90ba-41b2-a10a-72e917ae092e-plugins-config-volume\") pod \"b1122c44-90ba-41b2-a10a-72e917ae092e\" (UID: \"b1122c44-90ba-41b2-a10a-72e917ae092e\") " Nov 02 02:08:13.803242 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:13.803121 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1122c44-90ba-41b2-a10a-72e917ae092e-plugins-config-volume" pod "b1122c44-90ba-41b2-a10a-72e917ae092e" (UID: "b1122c44-90ba-41b2-a10a-72e917ae092e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:08:13.805006 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:13.804977 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1122c44-90ba-41b2-a10a-72e917ae092e-kube-api-access-567d8" pod "b1122c44-90ba-41b2-a10a-72e917ae092e" (UID: "b1122c44-90ba-41b2-a10a-72e917ae092e"). InnerVolumeSpecName "kube-api-access-567d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:08:13.904137 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:13.904109 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85abf3dc-3eea-41cc-bbaf-2d0208adc731-kube-api-access-qkcrz\" (UniqueName: \"kubernetes.io/projected/85abf3dc-3eea-41cc-bbaf-2d0208adc731-kube-api-access-qkcrz\") pod \"85abf3dc-3eea-41cc-bbaf-2d0208adc731\" (UID: \"85abf3dc-3eea-41cc-bbaf-2d0208adc731\") " Nov 02 02:08:13.904291 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:13.904208 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85abf3dc-3eea-41cc-bbaf-2d0208adc731-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85abf3dc-3eea-41cc-bbaf-2d0208adc731-plugins-config-volume\") pod \"85abf3dc-3eea-41cc-bbaf-2d0208adc731\" (UID: \"85abf3dc-3eea-41cc-bbaf-2d0208adc731\") " Nov 02 02:08:13.904455 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:13.904437 5044 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1122c44-90ba-41b2-a10a-72e917ae092e-plugins-config-volume\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:08:13.904455 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:13.904455 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-567d8\" (UniqueName: \"kubernetes.io/projected/b1122c44-90ba-41b2-a10a-72e917ae092e-kube-api-access-567d8\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:08:13.904622 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:13.904532 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85abf3dc-3eea-41cc-bbaf-2d0208adc731-plugins-config-volume" pod "85abf3dc-3eea-41cc-bbaf-2d0208adc731" (UID: "85abf3dc-3eea-41cc-bbaf-2d0208adc731"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:08:13.906196 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:13.906177 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85abf3dc-3eea-41cc-bbaf-2d0208adc731-kube-api-access-qkcrz" pod "85abf3dc-3eea-41cc-bbaf-2d0208adc731" (UID: "85abf3dc-3eea-41cc-bbaf-2d0208adc731"). InnerVolumeSpecName "kube-api-access-qkcrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:08:14.006027 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:14.005945 5044 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85abf3dc-3eea-41cc-bbaf-2d0208adc731-plugins-config-volume\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:08:14.006027 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:14.005970 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkcrz\" (UniqueName: \"kubernetes.io/projected/85abf3dc-3eea-41cc-bbaf-2d0208adc731-kube-api-access-qkcrz\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:08:14.009573 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:14.009546 5044 generic.go:356] "Generic (PLEG): container finished" podID="b1122c44-90ba-41b2-a10a-72e917ae092e" containerID="56dff402c110ad69e57c4e1ef97105f76a69fbeb052f8e40e1ac329a43e39714" exitCode=0 Nov 02 02:08:14.009715 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:14.009589 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26" Nov 02 02:08:14.009715 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:14.009629 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26" event={"ID":"b1122c44-90ba-41b2-a10a-72e917ae092e","Type":"ContainerDied","Data":"56dff402c110ad69e57c4e1ef97105f76a69fbeb052f8e40e1ac329a43e39714"} Nov 02 02:08:14.009834 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:14.009713 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26" event={"ID":"b1122c44-90ba-41b2-a10a-72e917ae092e","Type":"ContainerDied","Data":"1b004f6675b7d81d857d212bc9fb99983b7542aff930213f864b3e360cbf077e"} Nov 02 02:08:14.009834 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:14.009738 5044 scope.go:122] "RemoveContainer" containerID="56dff402c110ad69e57c4e1ef97105f76a69fbeb052f8e40e1ac329a43e39714" Nov 02 02:08:14.011247 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:14.011220 5044 generic.go:356] "Generic (PLEG): container finished" podID="85abf3dc-3eea-41cc-bbaf-2d0208adc731" containerID="dbbd4c2cf96fc8ce0fdce552c0f0b4d62c4f4df932a577fd716143281d560d14" exitCode=0 Nov 02 02:08:14.011371 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:14.011261 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42" Nov 02 02:08:14.011371 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:14.011300 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42" event={"ID":"85abf3dc-3eea-41cc-bbaf-2d0208adc731","Type":"ContainerDied","Data":"dbbd4c2cf96fc8ce0fdce552c0f0b4d62c4f4df932a577fd716143281d560d14"} Nov 02 02:08:14.011371 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:14.011342 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42" event={"ID":"85abf3dc-3eea-41cc-bbaf-2d0208adc731","Type":"ContainerDied","Data":"38d27b05848dfff5d644734672181b62845298fb892e9de948f5a2ba29febbd1"} Nov 02 02:08:14.020645 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:14.020626 5044 scope.go:122] "RemoveContainer" containerID="56dff402c110ad69e57c4e1ef97105f76a69fbeb052f8e40e1ac329a43e39714" Nov 02 02:08:14.020984 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:08:14.020965 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56dff402c110ad69e57c4e1ef97105f76a69fbeb052f8e40e1ac329a43e39714\": container with ID starting with 56dff402c110ad69e57c4e1ef97105f76a69fbeb052f8e40e1ac329a43e39714 not found: ID does not exist" containerID="56dff402c110ad69e57c4e1ef97105f76a69fbeb052f8e40e1ac329a43e39714" Nov 02 02:08:14.021056 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:14.020991 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56dff402c110ad69e57c4e1ef97105f76a69fbeb052f8e40e1ac329a43e39714"} err="failed to get container status \"56dff402c110ad69e57c4e1ef97105f76a69fbeb052f8e40e1ac329a43e39714\": rpc error: code = NotFound desc = could not find container \"56dff402c110ad69e57c4e1ef97105f76a69fbeb052f8e40e1ac329a43e39714\": container with ID starting with 56dff402c110ad69e57c4e1ef97105f76a69fbeb052f8e40e1ac329a43e39714 not found: ID does not exist" Nov 02 02:08:14.021056 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:14.021006 5044 scope.go:122] "RemoveContainer" containerID="dbbd4c2cf96fc8ce0fdce552c0f0b4d62c4f4df932a577fd716143281d560d14" Nov 02 02:08:14.029622 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:14.029604 5044 scope.go:122] "RemoveContainer" containerID="dbbd4c2cf96fc8ce0fdce552c0f0b4d62c4f4df932a577fd716143281d560d14" Nov 02 02:08:14.029898 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:08:14.029878 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbbd4c2cf96fc8ce0fdce552c0f0b4d62c4f4df932a577fd716143281d560d14\": container with ID starting with dbbd4c2cf96fc8ce0fdce552c0f0b4d62c4f4df932a577fd716143281d560d14 not found: ID does not exist" containerID="dbbd4c2cf96fc8ce0fdce552c0f0b4d62c4f4df932a577fd716143281d560d14" Nov 02 02:08:14.029969 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:14.029905 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbd4c2cf96fc8ce0fdce552c0f0b4d62c4f4df932a577fd716143281d560d14"} err="failed to get container status \"dbbd4c2cf96fc8ce0fdce552c0f0b4d62c4f4df932a577fd716143281d560d14\": rpc error: code = NotFound desc = could not find container \"dbbd4c2cf96fc8ce0fdce552c0f0b4d62c4f4df932a577fd716143281d560d14\": container with ID starting with dbbd4c2cf96fc8ce0fdce552c0f0b4d62c4f4df932a577fd716143281d560d14 not found: ID does not exist" Nov 02 02:08:14.178397 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:14.178355 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26"] Nov 02 02:08:14.183715 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:14.183689 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-76e530-7d76dc4d99-86k26"] Nov 02 02:08:14.295129 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:14.295100 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42"] Nov 02 02:08:14.299562 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:14.299538 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-76e530-57cd8df79f-fll42"] Nov 02 02:08:15.454864 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:15.454828 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85abf3dc-3eea-41cc-bbaf-2d0208adc731" path="/var/lib/kubelet/pods/85abf3dc-3eea-41cc-bbaf-2d0208adc731/volumes" Nov 02 02:08:15.455372 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:15.455357 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1122c44-90ba-41b2-a10a-72e917ae092e" path="/var/lib/kubelet/pods/b1122c44-90ba-41b2-a10a-72e917ae092e/volumes" Nov 02 02:08:16.779494 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:16.779455 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:21.580743 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:21.580697 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:24.174194 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:08:24.174150 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:08:25.806754 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:25.806719 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:27.346478 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:27.346446 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:27.575766 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:27.575734 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:27.722076 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:27.721975 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:28.402049 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:28.402004 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:30.397654 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:30.397620 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:32.078544 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:32.078512 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:33.649550 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:33.649515 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:33.863430 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:33.863396 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:34.012302 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:34.012267 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:34.585287 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:34.585251 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:34.829832 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:34.829801 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:34.983630 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:34.983541 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:35.157106 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:35.157073 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:39.017941 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:39.017910 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:41.034038 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:41.034004 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:41.179183 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:41.179146 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:41.409283 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:41.409213 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:44.129811 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:44.129775 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:48.214044 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:48.213974 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:49.312155 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:49.312107 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:53.788780 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:53.788511 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt"] Nov 02 02:08:54.037901 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:54.037868 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:54.172787 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:08:54.172708 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:08:54.177478 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:54.177454 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:54.495354 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:54.495280 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:08:58.789684 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:58.789618 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt" podUID="2e26e23b-5900-423c-9e14-7122ce904c81" containerName="payload-pre-processing" containerID="cri-o://79df2b34d3ea4ca0a623369c1454d510cf86b19041dfadddfb8265fed3593b94" gracePeriod=25 Nov 02 02:08:59.075760 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:59.075730 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt" Nov 02 02:08:59.149062 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:59.149031 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e26e23b-5900-423c-9e14-7122ce904c81-kube-api-access-zj5z8\" (UniqueName: \"kubernetes.io/projected/2e26e23b-5900-423c-9e14-7122ce904c81-kube-api-access-zj5z8\") pod \"2e26e23b-5900-423c-9e14-7122ce904c81\" (UID: \"2e26e23b-5900-423c-9e14-7122ce904c81\") " Nov 02 02:08:59.149234 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:59.149222 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e26e23b-5900-423c-9e14-7122ce904c81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e26e23b-5900-423c-9e14-7122ce904c81-plugins-config-volume\") pod \"2e26e23b-5900-423c-9e14-7122ce904c81\" (UID: \"2e26e23b-5900-423c-9e14-7122ce904c81\") " Nov 02 02:08:59.149608 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:59.149577 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e26e23b-5900-423c-9e14-7122ce904c81-plugins-config-volume" pod "2e26e23b-5900-423c-9e14-7122ce904c81" (UID: "2e26e23b-5900-423c-9e14-7122ce904c81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:08:59.151061 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:59.151037 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e26e23b-5900-423c-9e14-7122ce904c81-kube-api-access-zj5z8" pod "2e26e23b-5900-423c-9e14-7122ce904c81" (UID: "2e26e23b-5900-423c-9e14-7122ce904c81"). InnerVolumeSpecName "kube-api-access-zj5z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:08:59.250252 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:59.250210 5044 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e26e23b-5900-423c-9e14-7122ce904c81-plugins-config-volume\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:08:59.250252 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:59.250249 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj5z8\" (UniqueName: \"kubernetes.io/projected/2e26e23b-5900-423c-9e14-7122ce904c81-kube-api-access-zj5z8\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:08:59.307321 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:59.307245 5044 generic.go:356] "Generic (PLEG): container finished" podID="2e26e23b-5900-423c-9e14-7122ce904c81" containerID="79df2b34d3ea4ca0a623369c1454d510cf86b19041dfadddfb8265fed3593b94" exitCode=0 Nov 02 02:08:59.307321 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:59.307311 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt" Nov 02 02:08:59.307503 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:59.307334 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt" event={"ID":"2e26e23b-5900-423c-9e14-7122ce904c81","Type":"ContainerDied","Data":"79df2b34d3ea4ca0a623369c1454d510cf86b19041dfadddfb8265fed3593b94"} Nov 02 02:08:59.307503 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:59.307398 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt" event={"ID":"2e26e23b-5900-423c-9e14-7122ce904c81","Type":"ContainerDied","Data":"1b52f46838f2ad9b1186ec8d3c79bfe93c07ce0fb5886bd0b1c7c60813b05e50"} Nov 02 02:08:59.307503 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:59.307426 5044 scope.go:122] "RemoveContainer" containerID="79df2b34d3ea4ca0a623369c1454d510cf86b19041dfadddfb8265fed3593b94" Nov 02 02:08:59.318257 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:59.318240 5044 scope.go:122] "RemoveContainer" containerID="79df2b34d3ea4ca0a623369c1454d510cf86b19041dfadddfb8265fed3593b94" Nov 02 02:08:59.318540 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:08:59.318521 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79df2b34d3ea4ca0a623369c1454d510cf86b19041dfadddfb8265fed3593b94\": container with ID starting with 79df2b34d3ea4ca0a623369c1454d510cf86b19041dfadddfb8265fed3593b94 not found: ID does not exist" containerID="79df2b34d3ea4ca0a623369c1454d510cf86b19041dfadddfb8265fed3593b94" Nov 02 02:08:59.318594 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:59.318547 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79df2b34d3ea4ca0a623369c1454d510cf86b19041dfadddfb8265fed3593b94"} err="failed to get container status \"79df2b34d3ea4ca0a623369c1454d510cf86b19041dfadddfb8265fed3593b94\": rpc error: code = NotFound desc = could not find container \"79df2b34d3ea4ca0a623369c1454d510cf86b19041dfadddfb8265fed3593b94\": container with ID starting with 79df2b34d3ea4ca0a623369c1454d510cf86b19041dfadddfb8265fed3593b94 not found: ID does not exist" Nov 02 02:08:59.477728 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:59.477665 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt"] Nov 02 02:08:59.481725 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:59.481695 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c78625-58bb55797f-cf8tt"] Nov 02 02:08:59.585218 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:08:59.585123 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:01.452100 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:01.452058 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e26e23b-5900-423c-9e14-7122ce904c81" path="/var/lib/kubelet/pods/2e26e23b-5900-423c-9e14-7122ce904c81/volumes" Nov 02 02:09:04.576466 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:04.576423 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:09.288019 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:09.287978 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:14.103112 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:14.103078 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:18.159093 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:18.159056 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5"] Nov 02 02:09:18.159580 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:18.159309 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" podUID="9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d" containerName="maas-api" containerID="cri-o://7488323eda2119b38eb132538a3f76c6e5162aba91941fed1b65e6d526e14565" gracePeriod=30 Nov 02 02:09:18.252753 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:18.252700 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:18.426562 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:18.426467 5044 generic.go:356] "Generic (PLEG): container finished" podID="9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d" containerID="7488323eda2119b38eb132538a3f76c6e5162aba91941fed1b65e6d526e14565" exitCode=0 Nov 02 02:09:18.426562 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:18.426530 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" event={"ID":"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d","Type":"ContainerDied","Data":"7488323eda2119b38eb132538a3f76c6e5162aba91941fed1b65e6d526e14565"} Nov 02 02:09:18.467315 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:18.467289 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" Nov 02 02:09:18.467526 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:18.467500 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-kube-api-access-kvfll\" (UniqueName: \"kubernetes.io/projected/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-kube-api-access-kvfll\") pod \"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d\" (UID: \"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d\") " Nov 02 02:09:18.467619 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:18.467604 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-maas-api-tls\") pod \"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d\" (UID: \"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d\") " Nov 02 02:09:18.467852 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:18.467835 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-metrics-certs\") pod \"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d\" (UID: \"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d\") " Nov 02 02:09:18.467925 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:18.467871 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-openshift-service-ca\") pod \"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d\" (UID: \"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d\") " Nov 02 02:09:18.468322 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:18.468293 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-openshift-service-ca" pod "9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d" (UID: "9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:09:18.469683 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:18.469646 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-maas-api-tls" pod "9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d" (UID: "9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:09:18.469789 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:18.469739 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-kube-api-access-kvfll" pod "9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d" (UID: "9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d"). InnerVolumeSpecName "kube-api-access-kvfll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:09:18.469892 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:18.469867 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-metrics-certs" pod "9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d" (UID: "9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:09:18.569256 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:18.569156 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-maas-api-tls\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:09:18.569256 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:18.569196 5044 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-metrics-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:09:18.569256 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:18.569212 5044 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-openshift-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:09:18.569256 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:18.569225 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvfll\" (UniqueName: \"kubernetes.io/projected/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d-kube-api-access-kvfll\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:09:19.434457 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:19.434425 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" Nov 02 02:09:19.434983 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:19.434459 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5" event={"ID":"9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d","Type":"ContainerDied","Data":"69169ec23e823aee774288f0e4e164833501b7d12073495ce641e44edffbb885"} Nov 02 02:09:19.434983 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:19.434522 5044 scope.go:122] "RemoveContainer" containerID="7488323eda2119b38eb132538a3f76c6e5162aba91941fed1b65e6d526e14565" Nov 02 02:09:19.603957 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:19.603921 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5"] Nov 02 02:09:19.654904 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:19.654874 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-76e530-766d49b76f-pgsl5"] Nov 02 02:09:19.752836 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:19.752756 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:20.029297 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:20.029262 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:20.174410 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:20.174372 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:20.745071 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:20.745024 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:21.454034 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:21.453996 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d" path="/var/lib/kubelet/pods/9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d/volumes" Nov 02 02:09:24.173396 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:09:24.173355 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:09:24.974815 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:24.974777 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:26.755360 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:26.755320 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:26.899059 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:26.899005 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:27.098896 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:27.098867 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:30.009603 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:30.009567 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:35.008183 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:35.008149 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:39.452587 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:39.452553 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9"] Nov 02 02:09:39.813014 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:39.812979 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:39.957338 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:39.957301 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:43.808992 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:43.808963 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 02:09:43.815074 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:43.815048 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 02:09:44.453190 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:44.453153 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9" podUID="581a88ec-7e87-4e19-b6d8-093d7f2b2cb7" containerName="payload-processing" containerID="cri-o://11eefe167b370108aeb1b94c71d1ff6d11f73537a78be68f025a690b47185e24" gracePeriod=25 Nov 02 02:09:44.600353 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:44.600317 5044 generic.go:356] "Generic (PLEG): container finished" podID="581a88ec-7e87-4e19-b6d8-093d7f2b2cb7" containerID="11eefe167b370108aeb1b94c71d1ff6d11f73537a78be68f025a690b47185e24" exitCode=0 Nov 02 02:09:44.600489 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:44.600443 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9" event={"ID":"581a88ec-7e87-4e19-b6d8-093d7f2b2cb7","Type":"ContainerDied","Data":"11eefe167b370108aeb1b94c71d1ff6d11f73537a78be68f025a690b47185e24"} Nov 02 02:09:44.695101 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:44.695066 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:44.789221 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:44.789191 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9" Nov 02 02:09:44.891099 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:44.891061 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/581a88ec-7e87-4e19-b6d8-093d7f2b2cb7-kube-api-access-5bt8d\" (UniqueName: \"kubernetes.io/projected/581a88ec-7e87-4e19-b6d8-093d7f2b2cb7-kube-api-access-5bt8d\") pod \"581a88ec-7e87-4e19-b6d8-093d7f2b2cb7\" (UID: \"581a88ec-7e87-4e19-b6d8-093d7f2b2cb7\") " Nov 02 02:09:44.891594 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:44.891141 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/581a88ec-7e87-4e19-b6d8-093d7f2b2cb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/581a88ec-7e87-4e19-b6d8-093d7f2b2cb7-plugins-config-volume\") pod \"581a88ec-7e87-4e19-b6d8-093d7f2b2cb7\" (UID: \"581a88ec-7e87-4e19-b6d8-093d7f2b2cb7\") " Nov 02 02:09:44.891594 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:44.891551 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/581a88ec-7e87-4e19-b6d8-093d7f2b2cb7-plugins-config-volume" pod "581a88ec-7e87-4e19-b6d8-093d7f2b2cb7" (UID: "581a88ec-7e87-4e19-b6d8-093d7f2b2cb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:09:44.893243 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:44.893210 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/581a88ec-7e87-4e19-b6d8-093d7f2b2cb7-kube-api-access-5bt8d" pod "581a88ec-7e87-4e19-b6d8-093d7f2b2cb7" (UID: "581a88ec-7e87-4e19-b6d8-093d7f2b2cb7"). InnerVolumeSpecName "kube-api-access-5bt8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:09:44.992634 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:44.992536 5044 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/581a88ec-7e87-4e19-b6d8-093d7f2b2cb7-plugins-config-volume\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:09:44.992634 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:44.992572 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bt8d\" (UniqueName: \"kubernetes.io/projected/581a88ec-7e87-4e19-b6d8-093d7f2b2cb7-kube-api-access-5bt8d\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:09:45.609138 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:45.609104 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9" event={"ID":"581a88ec-7e87-4e19-b6d8-093d7f2b2cb7","Type":"ContainerDied","Data":"973bde58a2efe3a1692ef206c2120c114ce2848a64afe3a2898d351973bf3f8d"} Nov 02 02:09:45.609335 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:45.609143 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9" Nov 02 02:09:45.609335 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:45.609166 5044 scope.go:122] "RemoveContainer" containerID="11eefe167b370108aeb1b94c71d1ff6d11f73537a78be68f025a690b47185e24" Nov 02 02:09:45.780362 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:45.780326 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9"] Nov 02 02:09:45.829557 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:45.829530 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9039bf-766b7c86ff-xjvm9"] Nov 02 02:09:46.468013 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:46.467980 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj"] Nov 02 02:09:46.468437 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:46.468167 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" podUID="652e50af-8a91-43ff-bb49-bbdf781feefd" containerName="istio-proxy" containerID="cri-o://3ce551f917477de5db7686e019549561678b7a87110d88adf8a607abc00f5320" gracePeriod=30 Nov 02 02:09:47.452192 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:47.452160 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="581a88ec-7e87-4e19-b6d8-093d7f2b2cb7" path="/var/lib/kubelet/pods/581a88ec-7e87-4e19-b6d8-093d7f2b2cb7/volumes" Nov 02 02:09:48.561809 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:48.561776 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:49.885641 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:49.885607 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:51.652965 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.652651 5044 generic.go:356] "Generic (PLEG): container finished" podID="652e50af-8a91-43ff-bb49-bbdf781feefd" containerID="3ce551f917477de5db7686e019549561678b7a87110d88adf8a607abc00f5320" exitCode=0 Nov 02 02:09:51.652965 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.652865 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" event={"ID":"652e50af-8a91-43ff-bb49-bbdf781feefd","Type":"ContainerDied","Data":"3ce551f917477de5db7686e019549561678b7a87110d88adf8a607abc00f5320"} Nov 02 02:09:51.652965 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.652935 5044 scope.go:122] "RemoveContainer" containerID="acd61ca1e138ca51879c99ddcee190bd2c8d06866d65c415320e2218eb3130a3" Nov 02 02:09:51.737394 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.737342 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 02:09:51.773746 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.773721 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-data\") pod \"652e50af-8a91-43ff-bb49-bbdf781feefd\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " Nov 02 02:09:51.773890 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.773764 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-token\" (UniqueName: \"kubernetes.io/projected/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-token\") pod \"652e50af-8a91-43ff-bb49-bbdf781feefd\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " Nov 02 02:09:51.773890 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.773795 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652e50af-8a91-43ff-bb49-bbdf781feefd-kube-api-access-8c8lq\" (UniqueName: \"kubernetes.io/projected/652e50af-8a91-43ff-bb49-bbdf781feefd-kube-api-access-8c8lq\") pod \"652e50af-8a91-43ff-bb49-bbdf781feefd\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " Nov 02 02:09:51.773890 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.773860 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-workload-socket\") pod \"652e50af-8a91-43ff-bb49-bbdf781feefd\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " Nov 02 02:09:51.774040 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.773917 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-podinfo\") pod \"652e50af-8a91-43ff-bb49-bbdf781feefd\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " Nov 02 02:09:51.774040 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.773953 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/652e50af-8a91-43ff-bb49-bbdf781feefd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652e50af-8a91-43ff-bb49-bbdf781feefd-istiod-ca-cert\") pod \"652e50af-8a91-43ff-bb49-bbdf781feefd\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " Nov 02 02:09:51.774040 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.774003 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-envoy\") pod \"652e50af-8a91-43ff-bb49-bbdf781feefd\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " Nov 02 02:09:51.774040 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.774035 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-workload-certs\") pod \"652e50af-8a91-43ff-bb49-bbdf781feefd\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " Nov 02 02:09:51.774254 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.774051 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-data" pod "652e50af-8a91-43ff-bb49-bbdf781feefd" (UID: "652e50af-8a91-43ff-bb49-bbdf781feefd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:09:51.774254 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.774167 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-credential-socket\") pod \"652e50af-8a91-43ff-bb49-bbdf781feefd\" (UID: \"652e50af-8a91-43ff-bb49-bbdf781feefd\") " Nov 02 02:09:51.774524 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.774454 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-workload-socket" pod "652e50af-8a91-43ff-bb49-bbdf781feefd" (UID: "652e50af-8a91-43ff-bb49-bbdf781feefd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:09:51.774524 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.774491 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/652e50af-8a91-43ff-bb49-bbdf781feefd-istiod-ca-cert" pod "652e50af-8a91-43ff-bb49-bbdf781feefd" (UID: "652e50af-8a91-43ff-bb49-bbdf781feefd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:09:51.774813 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.774699 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-credential-socket" pod "652e50af-8a91-43ff-bb49-bbdf781feefd" (UID: "652e50af-8a91-43ff-bb49-bbdf781feefd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:09:51.774813 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.774720 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-workload-certs" pod "652e50af-8a91-43ff-bb49-bbdf781feefd" (UID: "652e50af-8a91-43ff-bb49-bbdf781feefd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:09:51.774813 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.774790 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:09:51.774813 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.774814 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652e50af-8a91-43ff-bb49-bbdf781feefd-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:09:51.775018 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.774829 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:09:51.776638 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.776606 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-envoy" pod "652e50af-8a91-43ff-bb49-bbdf781feefd" (UID: "652e50af-8a91-43ff-bb49-bbdf781feefd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:09:51.776810 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.776735 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-token" pod "652e50af-8a91-43ff-bb49-bbdf781feefd" (UID: "652e50af-8a91-43ff-bb49-bbdf781feefd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:09:51.776955 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.776935 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-podinfo" pod "652e50af-8a91-43ff-bb49-bbdf781feefd" (UID: "652e50af-8a91-43ff-bb49-bbdf781feefd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 02:09:51.777145 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.777131 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652e50af-8a91-43ff-bb49-bbdf781feefd-kube-api-access-8c8lq" pod "652e50af-8a91-43ff-bb49-bbdf781feefd" (UID: "652e50af-8a91-43ff-bb49-bbdf781feefd"). InnerVolumeSpecName "kube-api-access-8c8lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:09:51.875417 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.875387 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:09:51.875417 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.875412 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c8lq\" (UniqueName: \"kubernetes.io/projected/652e50af-8a91-43ff-bb49-bbdf781feefd-kube-api-access-8c8lq\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:09:51.875417 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.875421 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:09:51.875722 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.875431 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:09:51.875722 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.875440 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:09:51.875722 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:51.875448 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652e50af-8a91-43ff-bb49-bbdf781feefd-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:09:52.659946 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:52.659918 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" Nov 02 02:09:52.660367 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:52.659956 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj" event={"ID":"652e50af-8a91-43ff-bb49-bbdf781feefd","Type":"ContainerDied","Data":"c1314c4b60307e69af2fcc8e3ff2fa8f5c983de81d52c1149f9058931924a5f3"} Nov 02 02:09:52.660367 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:52.660025 5044 scope.go:122] "RemoveContainer" containerID="3ce551f917477de5db7686e019549561678b7a87110d88adf8a607abc00f5320" Nov 02 02:09:52.879729 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:52.879693 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj"] Nov 02 02:09:52.930724 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:52.930630 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e06ea2-openshift-default-7bc5d4b9cc-stswj"] Nov 02 02:09:53.452850 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:53.452814 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="652e50af-8a91-43ff-bb49-bbdf781feefd" path="/var/lib/kubelet/pods/652e50af-8a91-43ff-bb49-bbdf781feefd/volumes" Nov 02 02:09:54.171392 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:09:54.171359 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:09:54.285356 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:54.285331 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:09:59.068445 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:09:59.068403 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:03.776835 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:03.776783 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:08.583874 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:08.583840 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:13.379043 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:13.379011 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:18.171888 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:18.171839 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:22.353048 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:22.353011 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:23.856413 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:23.856379 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:24.092832 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:24.092800 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:24.172351 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:10:24.172264 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:10:24.239634 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:24.239605 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:24.768905 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:24.768862 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:29.391606 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:29.391563 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:30.704547 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:30.704503 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:30.853155 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:30.853123 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:31.028510 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:31.028478 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4"] Nov 02 02:10:31.028760 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:31.028739 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" podUID="e16f31d3-d566-4620-a625-097e8ee9e28b" containerName="istio-proxy" containerID="cri-o://b21d5788b748885131e2d7156bf65dde524f023a4d6fe7e26df32203e4cdff56" gracePeriod=30 Nov 02 02:10:31.035888 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:31.035864 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:31.519430 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:31.519396 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:34.391518 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:34.391486 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:36.310796 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.310739 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:10:36.380633 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.380610 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16f31d3-d566-4620-a625-097e8ee9e28b-kube-api-access-bdc7t\" (UniqueName: \"kubernetes.io/projected/e16f31d3-d566-4620-a625-097e8ee9e28b-kube-api-access-bdc7t\") pod \"e16f31d3-d566-4620-a625-097e8ee9e28b\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " Nov 02 02:10:36.380813 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.380717 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-workload-certs\") pod \"e16f31d3-d566-4620-a625-097e8ee9e28b\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " Nov 02 02:10:36.380813 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.380746 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-data\") pod \"e16f31d3-d566-4620-a625-097e8ee9e28b\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " Nov 02 02:10:36.380813 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.380771 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-podinfo\") pod \"e16f31d3-d566-4620-a625-097e8ee9e28b\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " Nov 02 02:10:36.380813 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.380804 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-token\" (UniqueName: \"kubernetes.io/projected/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-token\") pod \"e16f31d3-d566-4620-a625-097e8ee9e28b\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " Nov 02 02:10:36.381040 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.380858 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-workload-socket\") pod \"e16f31d3-d566-4620-a625-097e8ee9e28b\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " Nov 02 02:10:36.381040 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.380954 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-credential-socket\") pod \"e16f31d3-d566-4620-a625-097e8ee9e28b\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " Nov 02 02:10:36.381040 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.380982 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e16f31d3-d566-4620-a625-097e8ee9e28b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16f31d3-d566-4620-a625-097e8ee9e28b-istiod-ca-cert\") pod \"e16f31d3-d566-4620-a625-097e8ee9e28b\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " Nov 02 02:10:36.381212 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.381049 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-envoy\") pod \"e16f31d3-d566-4620-a625-097e8ee9e28b\" (UID: \"e16f31d3-d566-4620-a625-097e8ee9e28b\") " Nov 02 02:10:36.381212 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.381065 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-data" pod "e16f31d3-d566-4620-a625-097e8ee9e28b" (UID: "e16f31d3-d566-4620-a625-097e8ee9e28b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:10:36.381318 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.381239 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-workload-socket" pod "e16f31d3-d566-4620-a625-097e8ee9e28b" (UID: "e16f31d3-d566-4620-a625-097e8ee9e28b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:10:36.381541 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.381300 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-credential-socket" pod "e16f31d3-d566-4620-a625-097e8ee9e28b" (UID: "e16f31d3-d566-4620-a625-097e8ee9e28b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:10:36.381541 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.381394 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16f31d3-d566-4620-a625-097e8ee9e28b-istiod-ca-cert" pod "e16f31d3-d566-4620-a625-097e8ee9e28b" (UID: "e16f31d3-d566-4620-a625-097e8ee9e28b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:10:36.381541 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.381486 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:10:36.381541 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.381520 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-workload-certs" pod "e16f31d3-d566-4620-a625-097e8ee9e28b" (UID: "e16f31d3-d566-4620-a625-097e8ee9e28b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:10:36.383330 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.383294 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-token" pod "e16f31d3-d566-4620-a625-097e8ee9e28b" (UID: "e16f31d3-d566-4620-a625-097e8ee9e28b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:10:36.383440 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.383379 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16f31d3-d566-4620-a625-097e8ee9e28b-kube-api-access-bdc7t" pod "e16f31d3-d566-4620-a625-097e8ee9e28b" (UID: "e16f31d3-d566-4620-a625-097e8ee9e28b"). InnerVolumeSpecName "kube-api-access-bdc7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:10:36.383758 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.383738 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-podinfo" pod "e16f31d3-d566-4620-a625-097e8ee9e28b" (UID: "e16f31d3-d566-4620-a625-097e8ee9e28b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 02:10:36.383894 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.383872 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-envoy" pod "e16f31d3-d566-4620-a625-097e8ee9e28b" (UID: "e16f31d3-d566-4620-a625-097e8ee9e28b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:10:36.482836 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.482799 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:10:36.482836 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.482826 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:10:36.483060 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.482886 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:10:36.483060 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.482903 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:10:36.483060 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.482907 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84"] Nov 02 02:10:36.483060 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.482918 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:10:36.483060 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.483027 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16f31d3-d566-4620-a625-097e8ee9e28b-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:10:36.483060 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.483045 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16f31d3-d566-4620-a625-097e8ee9e28b-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:10:36.483060 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.483061 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdc7t\" (UniqueName: \"kubernetes.io/projected/e16f31d3-d566-4620-a625-097e8ee9e28b-kube-api-access-bdc7t\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:10:36.644723 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.644634 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:36.927203 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.927122 5044 generic.go:356] "Generic (PLEG): container finished" podID="e16f31d3-d566-4620-a625-097e8ee9e28b" containerID="b21d5788b748885131e2d7156bf65dde524f023a4d6fe7e26df32203e4cdff56" exitCode=0 Nov 02 02:10:36.927348 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.927202 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" Nov 02 02:10:36.927348 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.927202 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" event={"ID":"e16f31d3-d566-4620-a625-097e8ee9e28b","Type":"ContainerDied","Data":"b21d5788b748885131e2d7156bf65dde524f023a4d6fe7e26df32203e4cdff56"} Nov 02 02:10:36.927348 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.927260 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4" event={"ID":"e16f31d3-d566-4620-a625-097e8ee9e28b","Type":"ContainerDied","Data":"35c75f765cb763797606bab41c2657eec02c455a8855c18bd594788790f09eb9"} Nov 02 02:10:36.927348 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.927273 5044 scope.go:122] "RemoveContainer" containerID="b21d5788b748885131e2d7156bf65dde524f023a4d6fe7e26df32203e4cdff56" Nov 02 02:10:36.938269 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.938252 5044 scope.go:122] "RemoveContainer" containerID="a4428ddeda1e2cd7c8ca84a7f306996923b45e4d6282509ad6c62c4076c5bcd7" Nov 02 02:10:36.950175 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.950154 5044 scope.go:122] "RemoveContainer" containerID="b21d5788b748885131e2d7156bf65dde524f023a4d6fe7e26df32203e4cdff56" Nov 02 02:10:36.951008 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:10:36.950987 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b21d5788b748885131e2d7156bf65dde524f023a4d6fe7e26df32203e4cdff56\": container with ID starting with b21d5788b748885131e2d7156bf65dde524f023a4d6fe7e26df32203e4cdff56 not found: ID does not exist" containerID="b21d5788b748885131e2d7156bf65dde524f023a4d6fe7e26df32203e4cdff56" Nov 02 02:10:36.951107 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.951016 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b21d5788b748885131e2d7156bf65dde524f023a4d6fe7e26df32203e4cdff56"} err="failed to get container status \"b21d5788b748885131e2d7156bf65dde524f023a4d6fe7e26df32203e4cdff56\": rpc error: code = NotFound desc = could not find container \"b21d5788b748885131e2d7156bf65dde524f023a4d6fe7e26df32203e4cdff56\": container with ID starting with b21d5788b748885131e2d7156bf65dde524f023a4d6fe7e26df32203e4cdff56 not found: ID does not exist" Nov 02 02:10:36.951107 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.951041 5044 scope.go:122] "RemoveContainer" containerID="a4428ddeda1e2cd7c8ca84a7f306996923b45e4d6282509ad6c62c4076c5bcd7" Nov 02 02:10:36.951338 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:10:36.951321 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4428ddeda1e2cd7c8ca84a7f306996923b45e4d6282509ad6c62c4076c5bcd7\": container with ID starting with a4428ddeda1e2cd7c8ca84a7f306996923b45e4d6282509ad6c62c4076c5bcd7 not found: ID does not exist" containerID="a4428ddeda1e2cd7c8ca84a7f306996923b45e4d6282509ad6c62c4076c5bcd7" Nov 02 02:10:36.951384 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:36.951345 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4428ddeda1e2cd7c8ca84a7f306996923b45e4d6282509ad6c62c4076c5bcd7"} err="failed to get container status \"a4428ddeda1e2cd7c8ca84a7f306996923b45e4d6282509ad6c62c4076c5bcd7\": rpc error: code = NotFound desc = could not find container \"a4428ddeda1e2cd7c8ca84a7f306996923b45e4d6282509ad6c62c4076c5bcd7\": container with ID starting with a4428ddeda1e2cd7c8ca84a7f306996923b45e4d6282509ad6c62c4076c5bcd7 not found: ID does not exist" Nov 02 02:10:37.100147 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:37.100115 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4"] Nov 02 02:10:37.149295 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:37.149267 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c78625-openshift-default-7b95ffdd45-552c4"] Nov 02 02:10:37.452874 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:37.452840 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e16f31d3-d566-4620-a625-097e8ee9e28b" path="/var/lib/kubelet/pods/e16f31d3-d566-4620-a625-097e8ee9e28b/volumes" Nov 02 02:10:39.588297 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:39.588269 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:43.564342 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:43.564308 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l"] Nov 02 02:10:43.680261 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:43.680227 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d"] Nov 02 02:10:43.933595 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:43.933514 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:44.080307 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:44.080274 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:44.602776 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:44.602742 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:48.565552 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:48.565509 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l" podUID="d7d1b5a4-50ae-46d5-83f9-252cb2e8734c" containerName="payload-processing" containerID="cri-o://26e6bb3a3bfda4dc8a52a5c0eb2d8290eb216a96ddfe919f6ca282d7f9a37474" gracePeriod=25 Nov 02 02:10:48.680833 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:48.680798 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d" podUID="0f4ffc26-b174-4586-aa5d-11b08755848a" containerName="payload-pre-processing" containerID="cri-o://fb06780a486173a602bc7d8011c3e6ae3b72cfa268f333214be6450bde7e8f12" gracePeriod=25 Nov 02 02:10:48.847441 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:48.847379 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l" Nov 02 02:10:48.908032 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:48.907988 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:48.931149 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:48.931121 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d1b5a4-50ae-46d5-83f9-252cb2e8734c-kube-api-access-cnb9l\" (UniqueName: \"kubernetes.io/projected/d7d1b5a4-50ae-46d5-83f9-252cb2e8734c-kube-api-access-cnb9l\") pod \"d7d1b5a4-50ae-46d5-83f9-252cb2e8734c\" (UID: \"d7d1b5a4-50ae-46d5-83f9-252cb2e8734c\") " Nov 02 02:10:48.931314 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:48.931195 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d1b5a4-50ae-46d5-83f9-252cb2e8734c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d1b5a4-50ae-46d5-83f9-252cb2e8734c-plugins-config-volume\") pod \"d7d1b5a4-50ae-46d5-83f9-252cb2e8734c\" (UID: \"d7d1b5a4-50ae-46d5-83f9-252cb2e8734c\") " Nov 02 02:10:48.931602 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:48.931580 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d1b5a4-50ae-46d5-83f9-252cb2e8734c-plugins-config-volume" pod "d7d1b5a4-50ae-46d5-83f9-252cb2e8734c" (UID: "d7d1b5a4-50ae-46d5-83f9-252cb2e8734c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:10:48.933571 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:48.933551 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d1b5a4-50ae-46d5-83f9-252cb2e8734c-kube-api-access-cnb9l" pod "d7d1b5a4-50ae-46d5-83f9-252cb2e8734c" (UID: "d7d1b5a4-50ae-46d5-83f9-252cb2e8734c"). InnerVolumeSpecName "kube-api-access-cnb9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:10:48.955983 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:48.955959 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d" Nov 02 02:10:48.999591 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:48.999549 5044 generic.go:356] "Generic (PLEG): container finished" podID="0f4ffc26-b174-4586-aa5d-11b08755848a" containerID="fb06780a486173a602bc7d8011c3e6ae3b72cfa268f333214be6450bde7e8f12" exitCode=0 Nov 02 02:10:48.999777 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:48.999593 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d" Nov 02 02:10:48.999777 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:48.999605 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d" event={"ID":"0f4ffc26-b174-4586-aa5d-11b08755848a","Type":"ContainerDied","Data":"fb06780a486173a602bc7d8011c3e6ae3b72cfa268f333214be6450bde7e8f12"} Nov 02 02:10:48.999777 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:48.999662 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d" event={"ID":"0f4ffc26-b174-4586-aa5d-11b08755848a","Type":"ContainerDied","Data":"569bd9f6ebce564993cc4cc95c10effea5af42cd22e73496bddcd69d698859c5"} Nov 02 02:10:48.999777 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:48.999703 5044 scope.go:122] "RemoveContainer" containerID="fb06780a486173a602bc7d8011c3e6ae3b72cfa268f333214be6450bde7e8f12" Nov 02 02:10:49.001300 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.001275 5044 generic.go:356] "Generic (PLEG): container finished" podID="d7d1b5a4-50ae-46d5-83f9-252cb2e8734c" containerID="26e6bb3a3bfda4dc8a52a5c0eb2d8290eb216a96ddfe919f6ca282d7f9a37474" exitCode=0 Nov 02 02:10:49.001428 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.001321 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l" Nov 02 02:10:49.001428 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.001316 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l" event={"ID":"d7d1b5a4-50ae-46d5-83f9-252cb2e8734c","Type":"ContainerDied","Data":"26e6bb3a3bfda4dc8a52a5c0eb2d8290eb216a96ddfe919f6ca282d7f9a37474"} Nov 02 02:10:49.001534 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.001454 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l" event={"ID":"d7d1b5a4-50ae-46d5-83f9-252cb2e8734c","Type":"ContainerDied","Data":"7390f611dc33b13b17c2254a8f24bfa3a8941b39fcdaf35459b7c8dd0730b870"} Nov 02 02:10:49.011495 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.010822 5044 scope.go:122] "RemoveContainer" containerID="fb06780a486173a602bc7d8011c3e6ae3b72cfa268f333214be6450bde7e8f12" Nov 02 02:10:49.011495 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:10:49.011275 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb06780a486173a602bc7d8011c3e6ae3b72cfa268f333214be6450bde7e8f12\": container with ID starting with fb06780a486173a602bc7d8011c3e6ae3b72cfa268f333214be6450bde7e8f12 not found: ID does not exist" containerID="fb06780a486173a602bc7d8011c3e6ae3b72cfa268f333214be6450bde7e8f12" Nov 02 02:10:49.011495 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.011311 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb06780a486173a602bc7d8011c3e6ae3b72cfa268f333214be6450bde7e8f12"} err="failed to get container status \"fb06780a486173a602bc7d8011c3e6ae3b72cfa268f333214be6450bde7e8f12\": rpc error: code = NotFound desc = could not find container \"fb06780a486173a602bc7d8011c3e6ae3b72cfa268f333214be6450bde7e8f12\": container with ID starting with fb06780a486173a602bc7d8011c3e6ae3b72cfa268f333214be6450bde7e8f12 not found: ID does not exist" Nov 02 02:10:49.011495 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.011338 5044 scope.go:122] "RemoveContainer" containerID="26e6bb3a3bfda4dc8a52a5c0eb2d8290eb216a96ddfe919f6ca282d7f9a37474" Nov 02 02:10:49.020203 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.020178 5044 scope.go:122] "RemoveContainer" containerID="26e6bb3a3bfda4dc8a52a5c0eb2d8290eb216a96ddfe919f6ca282d7f9a37474" Nov 02 02:10:49.020525 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:10:49.020501 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e6bb3a3bfda4dc8a52a5c0eb2d8290eb216a96ddfe919f6ca282d7f9a37474\": container with ID starting with 26e6bb3a3bfda4dc8a52a5c0eb2d8290eb216a96ddfe919f6ca282d7f9a37474 not found: ID does not exist" containerID="26e6bb3a3bfda4dc8a52a5c0eb2d8290eb216a96ddfe919f6ca282d7f9a37474" Nov 02 02:10:49.020615 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.020529 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e6bb3a3bfda4dc8a52a5c0eb2d8290eb216a96ddfe919f6ca282d7f9a37474"} err="failed to get container status \"26e6bb3a3bfda4dc8a52a5c0eb2d8290eb216a96ddfe919f6ca282d7f9a37474\": rpc error: code = NotFound desc = could not find container \"26e6bb3a3bfda4dc8a52a5c0eb2d8290eb216a96ddfe919f6ca282d7f9a37474\": container with ID starting with 26e6bb3a3bfda4dc8a52a5c0eb2d8290eb216a96ddfe919f6ca282d7f9a37474 not found: ID does not exist" Nov 02 02:10:49.032261 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.032242 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f4ffc26-b174-4586-aa5d-11b08755848a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4ffc26-b174-4586-aa5d-11b08755848a-plugins-config-volume\") pod \"0f4ffc26-b174-4586-aa5d-11b08755848a\" (UID: \"0f4ffc26-b174-4586-aa5d-11b08755848a\") " Nov 02 02:10:49.032357 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.032277 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f4ffc26-b174-4586-aa5d-11b08755848a-kube-api-access-6hxcs\" (UniqueName: \"kubernetes.io/projected/0f4ffc26-b174-4586-aa5d-11b08755848a-kube-api-access-6hxcs\") pod \"0f4ffc26-b174-4586-aa5d-11b08755848a\" (UID: \"0f4ffc26-b174-4586-aa5d-11b08755848a\") " Nov 02 02:10:49.032564 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.032546 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4ffc26-b174-4586-aa5d-11b08755848a-plugins-config-volume" pod "0f4ffc26-b174-4586-aa5d-11b08755848a" (UID: "0f4ffc26-b174-4586-aa5d-11b08755848a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:10:49.032725 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.032703 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnb9l\" (UniqueName: \"kubernetes.io/projected/d7d1b5a4-50ae-46d5-83f9-252cb2e8734c-kube-api-access-cnb9l\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:10:49.032785 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.032733 5044 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d1b5a4-50ae-46d5-83f9-252cb2e8734c-plugins-config-volume\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:10:49.032785 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.032748 5044 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4ffc26-b174-4586-aa5d-11b08755848a-plugins-config-volume\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:10:49.034346 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.034324 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4ffc26-b174-4586-aa5d-11b08755848a-kube-api-access-6hxcs" pod "0f4ffc26-b174-4586-aa5d-11b08755848a" (UID: "0f4ffc26-b174-4586-aa5d-11b08755848a"). InnerVolumeSpecName "kube-api-access-6hxcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:10:49.134273 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.134194 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hxcs\" (UniqueName: \"kubernetes.io/projected/0f4ffc26-b174-4586-aa5d-11b08755848a-kube-api-access-6hxcs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:10:49.255142 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.255111 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l"] Nov 02 02:10:49.260200 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.260172 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-5888dc-78d6844599-q299l"] Nov 02 02:10:49.387543 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.387462 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:49.433878 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.433400 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d"] Nov 02 02:10:49.444186 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.442379 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-5888dc-78449ff74c-rlj2d"] Nov 02 02:10:49.459901 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.459870 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f4ffc26-b174-4586-aa5d-11b08755848a" path="/var/lib/kubelet/pods/0f4ffc26-b174-4586-aa5d-11b08755848a/volumes" Nov 02 02:10:49.460335 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.460314 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d1b5a4-50ae-46d5-83f9-252cb2e8734c" path="/var/lib/kubelet/pods/d7d1b5a4-50ae-46d5-83f9-252cb2e8734c/volumes" Nov 02 02:10:49.871595 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:49.871533 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:51.504148 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:51.504111 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" containerID="cri-o://15e823be3d44cd8f61bc792ddbe6dda0d7e51157a7611d562f610b2c72ce6402" gracePeriod=45 Nov 02 02:10:51.775012 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:51.774987 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 02:10:51.875921 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:51.875885 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-model-cache\") pod \"3ae3a465-c735-401b-a679-0a6141ca6bb9\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " Nov 02 02:10:51.875921 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:51.875921 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-home\" (UniqueName: \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-home\") pod \"3ae3a465-c735-401b-a679-0a6141ca6bb9\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " Nov 02 02:10:51.876198 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:51.875944 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-tmp-dir\") pod \"3ae3a465-c735-401b-a679-0a6141ca6bb9\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " Nov 02 02:10:51.876198 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:51.875975 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ae3a465-c735-401b-a679-0a6141ca6bb9-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ae3a465-c735-401b-a679-0a6141ca6bb9-tls-certs\") pod \"3ae3a465-c735-401b-a679-0a6141ca6bb9\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " Nov 02 02:10:51.876198 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:51.876073 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-dshm\") pod \"3ae3a465-c735-401b-a679-0a6141ca6bb9\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " Nov 02 02:10:51.876198 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:51.876156 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-home" pod "3ae3a465-c735-401b-a679-0a6141ca6bb9" (UID: "3ae3a465-c735-401b-a679-0a6141ca6bb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:10:51.876198 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:51.876172 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-model-cache" pod "3ae3a465-c735-401b-a679-0a6141ca6bb9" (UID: "3ae3a465-c735-401b-a679-0a6141ca6bb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:10:51.876484 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:51.876240 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae3a465-c735-401b-a679-0a6141ca6bb9-kube-api-access-lknq5\" (UniqueName: \"kubernetes.io/projected/3ae3a465-c735-401b-a679-0a6141ca6bb9-kube-api-access-lknq5\") pod \"3ae3a465-c735-401b-a679-0a6141ca6bb9\" (UID: \"3ae3a465-c735-401b-a679-0a6141ca6bb9\") " Nov 02 02:10:51.876484 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:51.876243 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-tmp-dir" pod "3ae3a465-c735-401b-a679-0a6141ca6bb9" (UID: "3ae3a465-c735-401b-a679-0a6141ca6bb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:10:51.876787 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:51.876752 5044 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-model-cache\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:10:51.876787 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:51.876773 5044 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-home\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:10:51.877011 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:51.876790 5044 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-tmp-dir\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:10:51.878471 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:51.878447 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae3a465-c735-401b-a679-0a6141ca6bb9-kube-api-access-lknq5" pod "3ae3a465-c735-401b-a679-0a6141ca6bb9" (UID: "3ae3a465-c735-401b-a679-0a6141ca6bb9"). InnerVolumeSpecName "kube-api-access-lknq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:10:51.879196 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:51.879177 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ae3a465-c735-401b-a679-0a6141ca6bb9-tls-certs" pod "3ae3a465-c735-401b-a679-0a6141ca6bb9" (UID: "3ae3a465-c735-401b-a679-0a6141ca6bb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:10:51.879498 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:51.879474 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-dshm" pod "3ae3a465-c735-401b-a679-0a6141ca6bb9" (UID: "3ae3a465-c735-401b-a679-0a6141ca6bb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:10:51.977656 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:51.977618 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lknq5\" (UniqueName: \"kubernetes.io/projected/3ae3a465-c735-401b-a679-0a6141ca6bb9-kube-api-access-lknq5\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:10:51.977656 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:51.977654 5044 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ae3a465-c735-401b-a679-0a6141ca6bb9-tls-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:10:51.977924 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:51.977664 5044 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ae3a465-c735-401b-a679-0a6141ca6bb9-dshm\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:10:52.021100 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:52.021069 5044 generic.go:356] "Generic (PLEG): container finished" podID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerID="15e823be3d44cd8f61bc792ddbe6dda0d7e51157a7611d562f610b2c72ce6402" exitCode=0 Nov 02 02:10:52.021279 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:52.021141 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" Nov 02 02:10:52.021279 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:52.021156 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" event={"ID":"3ae3a465-c735-401b-a679-0a6141ca6bb9","Type":"ContainerDied","Data":"15e823be3d44cd8f61bc792ddbe6dda0d7e51157a7611d562f610b2c72ce6402"} Nov 02 02:10:52.021279 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:52.021204 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84" event={"ID":"3ae3a465-c735-401b-a679-0a6141ca6bb9","Type":"ContainerDied","Data":"e7f45f7d7695365da3da18a8f0c8c1ae819a38a8bb22c6a5c8c4909242a1486a"} Nov 02 02:10:52.021279 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:52.021223 5044 scope.go:122] "RemoveContainer" containerID="15e823be3d44cd8f61bc792ddbe6dda0d7e51157a7611d562f610b2c72ce6402" Nov 02 02:10:52.032617 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:52.032596 5044 scope.go:122] "RemoveContainer" containerID="15e823be3d44cd8f61bc792ddbe6dda0d7e51157a7611d562f610b2c72ce6402" Nov 02 02:10:52.033269 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:10:52.033243 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e823be3d44cd8f61bc792ddbe6dda0d7e51157a7611d562f610b2c72ce6402\": container with ID starting with 15e823be3d44cd8f61bc792ddbe6dda0d7e51157a7611d562f610b2c72ce6402 not found: ID does not exist" containerID="15e823be3d44cd8f61bc792ddbe6dda0d7e51157a7611d562f610b2c72ce6402" Nov 02 02:10:52.033371 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:52.033290 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e823be3d44cd8f61bc792ddbe6dda0d7e51157a7611d562f610b2c72ce6402"} err="failed to get container status \"15e823be3d44cd8f61bc792ddbe6dda0d7e51157a7611d562f610b2c72ce6402\": rpc error: code = NotFound desc = could not find container \"15e823be3d44cd8f61bc792ddbe6dda0d7e51157a7611d562f610b2c72ce6402\": container with ID starting with 15e823be3d44cd8f61bc792ddbe6dda0d7e51157a7611d562f610b2c72ce6402 not found: ID does not exist" Nov 02 02:10:52.190099 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:52.190066 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84"] Nov 02 02:10:52.194068 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:52.194042 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5888dc/distinct-2-w6-5888dc-kserve-844bf46b48-7km84"] Nov 02 02:10:53.451480 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:53.451451 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" path="/var/lib/kubelet/pods/3ae3a465-c735-401b-a679-0a6141ca6bb9/volumes" Nov 02 02:10:54.176939 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:10:54.176894 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:10:54.183664 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:54.183636 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:58.360311 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:58.360285 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:10:59.861550 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:10:59.861513 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:00.097135 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:00.097100 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:00.244664 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:00.244583 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:00.816610 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:00.816573 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:05.171236 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:05.171197 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:06.941806 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:06.941773 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:07.087972 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:07.087933 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:07.281063 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:07.281020 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:07.732708 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:07.732608 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:10.304616 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:10.304583 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:12.709836 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:12.709804 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx"] Nov 02 02:11:12.812206 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:12.812170 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:13.946212 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:13.946178 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:14.103226 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:14.103197 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:14.469533 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:14.469503 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:15.286814 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:15.286784 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:20.051289 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:20.051257 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:20.199347 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:20.199313 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:20.746493 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:20.746456 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:20.894569 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:20.894535 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:24.173933 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:11:24.173874 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:11:25.180195 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:25.180147 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:27.731856 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:27.731818 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" containerID="cri-o://3727e355a31ffcf685f5885504d693553743c45d6e5372ea9d7b46e2031ab02e" gracePeriod=45 Nov 02 02:11:27.999047 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:27.998985 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 02:11:28.083051 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.083019 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-dshm\") pod \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " Nov 02 02:11:28.083240 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.083152 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-home\" (UniqueName: \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-home\") pod \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " Nov 02 02:11:28.083240 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.083175 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-kube-api-access-5bwll\" (UniqueName: \"kubernetes.io/projected/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-kube-api-access-5bwll\") pod \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " Nov 02 02:11:28.083240 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.083198 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-tmp-dir\") pod \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " Nov 02 02:11:28.083436 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.083272 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-model-cache\") pod \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " Nov 02 02:11:28.083436 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.083323 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-tls-certs\") pod \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\" (UID: \"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc\") " Nov 02 02:11:28.083436 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.083422 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-home" pod "cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" (UID: "cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:11:28.083605 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.083585 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-tmp-dir" pod "cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" (UID: "cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:11:28.083662 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.083601 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-model-cache" pod "cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" (UID: "cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:11:28.083873 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.083845 5044 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-model-cache\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:11:28.083873 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.083870 5044 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-home\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:11:28.084066 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.083885 5044 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-tmp-dir\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:11:28.085442 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.085418 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-dshm" pod "cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" (UID: "cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:11:28.085536 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.085460 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-tls-certs" pod "cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" (UID: "cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:11:28.085536 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.085515 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-kube-api-access-5bwll" pod "cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" (UID: "cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc"). InnerVolumeSpecName "kube-api-access-5bwll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:11:28.185651 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.185601 5044 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-dshm\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:11:28.185651 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.185641 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bwll\" (UniqueName: \"kubernetes.io/projected/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-kube-api-access-5bwll\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:11:28.185651 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.185658 5044 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc-tls-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:11:28.242980 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.242946 5044 generic.go:356] "Generic (PLEG): container finished" podID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerID="3727e355a31ffcf685f5885504d693553743c45d6e5372ea9d7b46e2031ab02e" exitCode=0 Nov 02 02:11:28.243166 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.242995 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" event={"ID":"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc","Type":"ContainerDied","Data":"3727e355a31ffcf685f5885504d693553743c45d6e5372ea9d7b46e2031ab02e"} Nov 02 02:11:28.243166 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.243002 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" Nov 02 02:11:28.243166 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.243028 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx" event={"ID":"cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc","Type":"ContainerDied","Data":"7955aac7fb88a6a1f006209dcd51b3f0c1324c525e0ae3d4eb8f670b59995514"} Nov 02 02:11:28.243166 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.243044 5044 scope.go:122] "RemoveContainer" containerID="3727e355a31ffcf685f5885504d693553743c45d6e5372ea9d7b46e2031ab02e" Nov 02 02:11:28.252724 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.252698 5044 scope.go:122] "RemoveContainer" containerID="3727e355a31ffcf685f5885504d693553743c45d6e5372ea9d7b46e2031ab02e" Nov 02 02:11:28.252997 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:11:28.252976 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3727e355a31ffcf685f5885504d693553743c45d6e5372ea9d7b46e2031ab02e\": container with ID starting with 3727e355a31ffcf685f5885504d693553743c45d6e5372ea9d7b46e2031ab02e not found: ID does not exist" containerID="3727e355a31ffcf685f5885504d693553743c45d6e5372ea9d7b46e2031ab02e" Nov 02 02:11:28.253073 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.253004 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3727e355a31ffcf685f5885504d693553743c45d6e5372ea9d7b46e2031ab02e"} err="failed to get container status \"3727e355a31ffcf685f5885504d693553743c45d6e5372ea9d7b46e2031ab02e\": rpc error: code = NotFound desc = could not find container \"3727e355a31ffcf685f5885504d693553743c45d6e5372ea9d7b46e2031ab02e\": container with ID starting with 3727e355a31ffcf685f5885504d693553743c45d6e5372ea9d7b46e2031ab02e not found: ID does not exist" Nov 02 02:11:28.410524 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.410489 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx"] Nov 02 02:11:28.416540 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:28.416511 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-88e76d/distinct-2-w3-88e76d-kserve-564d4cfbf9-969qx"] Nov 02 02:11:29.451532 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:29.451499 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" path="/var/lib/kubelet/pods/cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc/volumes" Nov 02 02:11:29.983388 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:29.983355 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:34.778544 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:34.778511 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:38.288240 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:38.288203 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:38.850257 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:38.850222 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f"] Nov 02 02:11:38.850491 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:38.850466 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" podUID="7812bf12-6286-4de7-88b4-b6d509c9482c" containerName="maas-api" containerID="cri-o://dcb5f8918141f533cf4b766953633bfa83292f32a9d8afdb75562b5d6478afd6" gracePeriod=30 Nov 02 02:11:38.957298 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:38.957267 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:39.160033 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.160007 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" Nov 02 02:11:39.310132 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.310095 5044 generic.go:356] "Generic (PLEG): container finished" podID="7812bf12-6286-4de7-88b4-b6d509c9482c" containerID="dcb5f8918141f533cf4b766953633bfa83292f32a9d8afdb75562b5d6478afd6" exitCode=0 Nov 02 02:11:39.310779 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.310166 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" Nov 02 02:11:39.310902 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.310176 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" event={"ID":"7812bf12-6286-4de7-88b4-b6d509c9482c","Type":"ContainerDied","Data":"dcb5f8918141f533cf4b766953633bfa83292f32a9d8afdb75562b5d6478afd6"} Nov 02 02:11:39.311001 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.310976 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7812bf12-6286-4de7-88b4-b6d509c9482c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7812bf12-6286-4de7-88b4-b6d509c9482c-maas-api-tls\") pod \"7812bf12-6286-4de7-88b4-b6d509c9482c\" (UID: \"7812bf12-6286-4de7-88b4-b6d509c9482c\") " Nov 02 02:11:39.311077 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.310997 5044 scope.go:122] "RemoveContainer" containerID="dcb5f8918141f533cf4b766953633bfa83292f32a9d8afdb75562b5d6478afd6" Nov 02 02:11:39.311135 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.311109 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7812bf12-6286-4de7-88b4-b6d509c9482c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7812bf12-6286-4de7-88b4-b6d509c9482c-openshift-service-ca\") pod \"7812bf12-6286-4de7-88b4-b6d509c9482c\" (UID: \"7812bf12-6286-4de7-88b4-b6d509c9482c\") " Nov 02 02:11:39.311202 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.311173 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7812bf12-6286-4de7-88b4-b6d509c9482c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7812bf12-6286-4de7-88b4-b6d509c9482c-metrics-certs\") pod \"7812bf12-6286-4de7-88b4-b6d509c9482c\" (UID: \"7812bf12-6286-4de7-88b4-b6d509c9482c\") " Nov 02 02:11:39.311260 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.311242 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7812bf12-6286-4de7-88b4-b6d509c9482c-kube-api-access-qtpkb\" (UniqueName: \"kubernetes.io/projected/7812bf12-6286-4de7-88b4-b6d509c9482c-kube-api-access-qtpkb\") pod \"7812bf12-6286-4de7-88b4-b6d509c9482c\" (UID: \"7812bf12-6286-4de7-88b4-b6d509c9482c\") " Nov 02 02:11:39.311704 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.311649 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7812bf12-6286-4de7-88b4-b6d509c9482c-openshift-service-ca" pod "7812bf12-6286-4de7-88b4-b6d509c9482c" (UID: "7812bf12-6286-4de7-88b4-b6d509c9482c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:11:39.311840 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.310970 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f" event={"ID":"7812bf12-6286-4de7-88b4-b6d509c9482c","Type":"ContainerDied","Data":"a1940279cec282b250b0d32778558361a6aebbf8d8b3c6232a6cad1a1bb50ad7"} Nov 02 02:11:39.315174 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.315150 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7812bf12-6286-4de7-88b4-b6d509c9482c-metrics-certs" pod "7812bf12-6286-4de7-88b4-b6d509c9482c" (UID: "7812bf12-6286-4de7-88b4-b6d509c9482c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:11:39.316641 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.316617 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7812bf12-6286-4de7-88b4-b6d509c9482c-maas-api-tls" pod "7812bf12-6286-4de7-88b4-b6d509c9482c" (UID: "7812bf12-6286-4de7-88b4-b6d509c9482c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:11:39.318300 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.318236 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7812bf12-6286-4de7-88b4-b6d509c9482c-kube-api-access-qtpkb" pod "7812bf12-6286-4de7-88b4-b6d509c9482c" (UID: "7812bf12-6286-4de7-88b4-b6d509c9482c"). InnerVolumeSpecName "kube-api-access-qtpkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:11:39.342350 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.342319 5044 scope.go:122] "RemoveContainer" containerID="dcb5f8918141f533cf4b766953633bfa83292f32a9d8afdb75562b5d6478afd6" Nov 02 02:11:39.342772 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:11:39.342749 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcb5f8918141f533cf4b766953633bfa83292f32a9d8afdb75562b5d6478afd6\": container with ID starting with dcb5f8918141f533cf4b766953633bfa83292f32a9d8afdb75562b5d6478afd6 not found: ID does not exist" containerID="dcb5f8918141f533cf4b766953633bfa83292f32a9d8afdb75562b5d6478afd6" Nov 02 02:11:39.342861 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.342782 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcb5f8918141f533cf4b766953633bfa83292f32a9d8afdb75562b5d6478afd6"} err="failed to get container status \"dcb5f8918141f533cf4b766953633bfa83292f32a9d8afdb75562b5d6478afd6\": rpc error: code = NotFound desc = could not find container \"dcb5f8918141f533cf4b766953633bfa83292f32a9d8afdb75562b5d6478afd6\": container with ID starting with dcb5f8918141f533cf4b766953633bfa83292f32a9d8afdb75562b5d6478afd6 not found: ID does not exist" Nov 02 02:11:39.415646 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.415504 5044 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7812bf12-6286-4de7-88b4-b6d509c9482c-openshift-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:11:39.415646 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.415544 5044 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7812bf12-6286-4de7-88b4-b6d509c9482c-metrics-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:11:39.415646 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.415557 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtpkb\" (UniqueName: \"kubernetes.io/projected/7812bf12-6286-4de7-88b4-b6d509c9482c-kube-api-access-qtpkb\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:11:39.415646 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.415571 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7812bf12-6286-4de7-88b4-b6d509c9482c-maas-api-tls\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:11:39.728441 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.728367 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f"] Nov 02 02:11:39.799569 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:39.799533 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5888dc-9c467b59-6xv5f"] Nov 02 02:11:40.466170 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:40.466138 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:40.688447 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:40.688413 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:40.836911 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:40.836877 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:41.403907 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:41.403875 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:41.471862 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:41.471823 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7812bf12-6286-4de7-88b4-b6d509c9482c" path="/var/lib/kubelet/pods/7812bf12-6286-4de7-88b4-b6d509c9482c/volumes" Nov 02 02:11:45.788815 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:45.788786 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:47.647136 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:47.647106 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:47.793166 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:47.793129 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:48.022280 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:48.022248 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:48.175666 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:48.175628 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:49.249236 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:49.249204 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:50.461345 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:50.461315 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:50.609200 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:50.609161 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:54.173194 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:11:54.173152 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:11:55.385915 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:55.385883 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:56.529366 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:56.529334 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:11:59.550497 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:11:59.550464 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:01.066881 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:01.066848 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:01.291926 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:01.291889 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:01.440570 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:01.440489 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:01.992976 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:01.992942 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:06.594548 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:06.594511 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:06.790036 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:06.789989 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:08.243640 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:08.243606 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:08.401248 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:08.401178 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:08.603036 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:08.602999 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:09.759874 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:09.759843 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:11.570003 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:11.569974 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:16.595356 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:16.595320 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:21.365268 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:21.365232 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:21.920239 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:21.920196 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq"] Nov 02 02:12:22.086473 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:22.086439 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:22.264624 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:22.263914 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:22.507979 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:22.507944 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:24.176582 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:12:24.176544 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:12:26.180641 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:26.180603 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:26.920786 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:26.920745 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq" podUID="dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8" containerName="payload-pre-processing" containerID="cri-o://cc3c3898bb35306d643b460c592251a04bbc9d256e1bf7bbb7652bb952c9f78a" gracePeriod=25 Nov 02 02:12:31.002153 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:31.002123 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:32.367563 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:32.367526 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq" podUID="dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 02:12:35.784310 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:35.784276 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:40.578003 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:40.577970 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:42.367787 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:42.367752 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq" podUID="dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 02:12:44.704064 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:44.704026 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf"] Nov 02 02:12:44.704545 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:44.704270 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" podUID="6a0186e1-b5ca-4475-b1bd-0003168fa850" containerName="maas-api" containerID="cri-o://221b434a0fecf0566722b1eac8d2cf2a7bcc540915e69b343687f634dabe5f11" gracePeriod=30 Nov 02 02:12:44.802777 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:44.802742 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:44.988143 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:44.988118 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" Nov 02 02:12:45.182093 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.182061 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a0186e1-b5ca-4475-b1bd-0003168fa850-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0186e1-b5ca-4475-b1bd-0003168fa850-openshift-service-ca\") pod \"6a0186e1-b5ca-4475-b1bd-0003168fa850\" (UID: \"6a0186e1-b5ca-4475-b1bd-0003168fa850\") " Nov 02 02:12:45.182273 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.182133 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a0186e1-b5ca-4475-b1bd-0003168fa850-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a0186e1-b5ca-4475-b1bd-0003168fa850-metrics-certs\") pod \"6a0186e1-b5ca-4475-b1bd-0003168fa850\" (UID: \"6a0186e1-b5ca-4475-b1bd-0003168fa850\") " Nov 02 02:12:45.182273 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.182167 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a0186e1-b5ca-4475-b1bd-0003168fa850-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a0186e1-b5ca-4475-b1bd-0003168fa850-maas-api-tls\") pod \"6a0186e1-b5ca-4475-b1bd-0003168fa850\" (UID: \"6a0186e1-b5ca-4475-b1bd-0003168fa850\") " Nov 02 02:12:45.182273 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.182206 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0186e1-b5ca-4475-b1bd-0003168fa850-kube-api-access-9fqrs\" (UniqueName: \"kubernetes.io/projected/6a0186e1-b5ca-4475-b1bd-0003168fa850-kube-api-access-9fqrs\") pod \"6a0186e1-b5ca-4475-b1bd-0003168fa850\" (UID: \"6a0186e1-b5ca-4475-b1bd-0003168fa850\") " Nov 02 02:12:45.182505 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.182478 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0186e1-b5ca-4475-b1bd-0003168fa850-openshift-service-ca" pod "6a0186e1-b5ca-4475-b1bd-0003168fa850" (UID: "6a0186e1-b5ca-4475-b1bd-0003168fa850"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:12:45.182645 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.182627 5044 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0186e1-b5ca-4475-b1bd-0003168fa850-openshift-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:45.184405 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.184373 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0186e1-b5ca-4475-b1bd-0003168fa850-metrics-certs" pod "6a0186e1-b5ca-4475-b1bd-0003168fa850" (UID: "6a0186e1-b5ca-4475-b1bd-0003168fa850"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:12:45.184515 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.184427 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0186e1-b5ca-4475-b1bd-0003168fa850-maas-api-tls" pod "6a0186e1-b5ca-4475-b1bd-0003168fa850" (UID: "6a0186e1-b5ca-4475-b1bd-0003168fa850"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:12:45.184515 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.184435 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0186e1-b5ca-4475-b1bd-0003168fa850-kube-api-access-9fqrs" pod "6a0186e1-b5ca-4475-b1bd-0003168fa850" (UID: "6a0186e1-b5ca-4475-b1bd-0003168fa850"). InnerVolumeSpecName "kube-api-access-9fqrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:12:45.283227 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.283195 5044 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a0186e1-b5ca-4475-b1bd-0003168fa850-metrics-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:45.283227 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.283226 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a0186e1-b5ca-4475-b1bd-0003168fa850-maas-api-tls\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:45.283442 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.283237 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fqrs\" (UniqueName: \"kubernetes.io/projected/6a0186e1-b5ca-4475-b1bd-0003168fa850-kube-api-access-9fqrs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:45.691692 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.691593 5044 generic.go:356] "Generic (PLEG): container finished" podID="6a0186e1-b5ca-4475-b1bd-0003168fa850" containerID="221b434a0fecf0566722b1eac8d2cf2a7bcc540915e69b343687f634dabe5f11" exitCode=0 Nov 02 02:12:45.691692 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.691629 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" event={"ID":"6a0186e1-b5ca-4475-b1bd-0003168fa850","Type":"ContainerDied","Data":"221b434a0fecf0566722b1eac8d2cf2a7bcc540915e69b343687f634dabe5f11"} Nov 02 02:12:45.691692 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.691641 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" Nov 02 02:12:45.691692 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.691682 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf" event={"ID":"6a0186e1-b5ca-4475-b1bd-0003168fa850","Type":"ContainerDied","Data":"fa73b54226b98541d51f86ecf750f2516730af822f844dfdabbb7115df82dd88"} Nov 02 02:12:45.691980 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.691704 5044 scope.go:122] "RemoveContainer" containerID="221b434a0fecf0566722b1eac8d2cf2a7bcc540915e69b343687f634dabe5f11" Nov 02 02:12:45.701622 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.701600 5044 scope.go:122] "RemoveContainer" containerID="221b434a0fecf0566722b1eac8d2cf2a7bcc540915e69b343687f634dabe5f11" Nov 02 02:12:45.702001 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:12:45.701971 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"221b434a0fecf0566722b1eac8d2cf2a7bcc540915e69b343687f634dabe5f11\": container with ID starting with 221b434a0fecf0566722b1eac8d2cf2a7bcc540915e69b343687f634dabe5f11 not found: ID does not exist" containerID="221b434a0fecf0566722b1eac8d2cf2a7bcc540915e69b343687f634dabe5f11" Nov 02 02:12:45.702091 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.702013 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"221b434a0fecf0566722b1eac8d2cf2a7bcc540915e69b343687f634dabe5f11"} err="failed to get container status \"221b434a0fecf0566722b1eac8d2cf2a7bcc540915e69b343687f634dabe5f11\": rpc error: code = NotFound desc = could not find container \"221b434a0fecf0566722b1eac8d2cf2a7bcc540915e69b343687f634dabe5f11\": container with ID starting with 221b434a0fecf0566722b1eac8d2cf2a7bcc540915e69b343687f634dabe5f11 not found: ID does not exist" Nov 02 02:12:45.861459 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.861421 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf"] Nov 02 02:12:45.912618 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:45.912586 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d0fdb9-698fd7cd79-s52mf"] Nov 02 02:12:46.342922 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:46.342884 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:46.587567 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:46.587533 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:46.732463 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:46.732383 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:47.178801 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:47.178770 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq" Nov 02 02:12:47.203509 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:47.203479 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8-plugins-config-volume\") pod \"dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8\" (UID: \"dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8\") " Nov 02 02:12:47.203642 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:47.203585 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8-kube-api-access-tlntl\" (UniqueName: \"kubernetes.io/projected/dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8-kube-api-access-tlntl\") pod \"dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8\" (UID: \"dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8\") " Nov 02 02:12:47.203850 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:47.203828 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8-plugins-config-volume" pod "dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8" (UID: "dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:12:47.203982 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:47.203964 5044 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8-plugins-config-volume\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:47.205536 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:47.205517 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8-kube-api-access-tlntl" pod "dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8" (UID: "dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8"). InnerVolumeSpecName "kube-api-access-tlntl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:12:47.304649 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:47.304585 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlntl\" (UniqueName: \"kubernetes.io/projected/dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8-kube-api-access-tlntl\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:47.321286 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:47.321259 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:47.452304 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:47.452266 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a0186e1-b5ca-4475-b1bd-0003168fa850" path="/var/lib/kubelet/pods/6a0186e1-b5ca-4475-b1bd-0003168fa850/volumes" Nov 02 02:12:47.711009 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:47.710925 5044 generic.go:356] "Generic (PLEG): container finished" podID="dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8" containerID="cc3c3898bb35306d643b460c592251a04bbc9d256e1bf7bbb7652bb952c9f78a" exitCode=0 Nov 02 02:12:47.711009 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:47.710977 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq" Nov 02 02:12:47.711223 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:47.711007 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq" event={"ID":"dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8","Type":"ContainerDied","Data":"cc3c3898bb35306d643b460c592251a04bbc9d256e1bf7bbb7652bb952c9f78a"} Nov 02 02:12:47.711223 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:47.711068 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq" event={"ID":"dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8","Type":"ContainerDied","Data":"dfcbf8f30deffd59e230bca909300319ca13a20da298fe3d7fc3e795bf2afb8f"} Nov 02 02:12:47.711223 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:47.711092 5044 scope.go:122] "RemoveContainer" containerID="cc3c3898bb35306d643b460c592251a04bbc9d256e1bf7bbb7652bb952c9f78a" Nov 02 02:12:47.720238 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:47.720222 5044 scope.go:122] "RemoveContainer" containerID="cc3c3898bb35306d643b460c592251a04bbc9d256e1bf7bbb7652bb952c9f78a" Nov 02 02:12:47.720526 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:12:47.720504 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc3c3898bb35306d643b460c592251a04bbc9d256e1bf7bbb7652bb952c9f78a\": container with ID starting with cc3c3898bb35306d643b460c592251a04bbc9d256e1bf7bbb7652bb952c9f78a not found: ID does not exist" containerID="cc3c3898bb35306d643b460c592251a04bbc9d256e1bf7bbb7652bb952c9f78a" Nov 02 02:12:47.720569 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:47.720534 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3c3898bb35306d643b460c592251a04bbc9d256e1bf7bbb7652bb952c9f78a"} err="failed to get container status \"cc3c3898bb35306d643b460c592251a04bbc9d256e1bf7bbb7652bb952c9f78a\": rpc error: code = NotFound desc = could not find container \"cc3c3898bb35306d643b460c592251a04bbc9d256e1bf7bbb7652bb952c9f78a\": container with ID starting with cc3c3898bb35306d643b460c592251a04bbc9d256e1bf7bbb7652bb952c9f78a not found: ID does not exist" Nov 02 02:12:47.879663 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:47.879633 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq"] Nov 02 02:12:47.883754 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:47.883728 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d0fdb9-67677c9bff-hbzbq"] Nov 02 02:12:49.451663 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:49.451632 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8" path="/var/lib/kubelet/pods/dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8/volumes" Nov 02 02:12:49.591934 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:49.591898 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:51.790725 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:51.790664 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:53.011002 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:53.010966 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf"] Nov 02 02:12:53.011405 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:53.011252 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" podUID="7b500c78-94b3-46ff-a95f-abdc414c5f3b" containerName="istio-proxy" containerID="cri-o://4c692abd1cdf223ce8890af6517f2cb3e27de5a6567954233680a8b8d1d65999" gracePeriod=30 Nov 02 02:12:53.060201 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:53.060162 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b"] Nov 02 02:12:53.060400 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:53.060376 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" podUID="766dfcbb-afda-4f73-8b00-8c53e2fb45d5" containerName="istio-proxy" containerID="cri-o://fb0067c9c9529e478e01367b5810cc450623aaaac89f0e769248a9c38dcd4fd7" gracePeriod=30 Nov 02 02:12:53.062323 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:53.062298 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk"] Nov 02 02:12:53.062521 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:53.062501 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" podUID="da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" containerName="istio-proxy" containerID="cri-o://5d246185e91e96a8a1e5dddd099a6b961167a701338521d5548af550570d7ae5" gracePeriod=30 Nov 02 02:12:53.112214 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:53.112180 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj"] Nov 02 02:12:53.112437 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:53.112413 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" podUID="e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" containerName="istio-proxy" containerID="cri-o://50b67d2dd2612e3b9b0c3f71d45475217ef8439fe8266bb5f4645e15cd2b6df2" gracePeriod=30 Nov 02 02:12:53.781002 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:53.780967 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:53.939635 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:53.939602 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:54.113558 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:54.113475 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:54.203702 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:12:54.202533 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:12:55.408306 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:55.408271 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 02:12:55.408697 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:55.408338 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" podUID="766dfcbb-afda-4f73-8b00-8c53e2fb45d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 02:12:55.956556 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:55.956520 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-pjltk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 02:12:55.956749 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:55.956590 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" podUID="da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 02:12:56.640045 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:56.640008 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 02:12:56.640444 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:56.640074 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" podUID="7b500c78-94b3-46ff-a95f-abdc414c5f3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 02:12:56.709286 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:56.709256 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:58.351448 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.351421 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 02:12:58.419794 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.419770 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 02:12:58.421653 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.421628 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-workload-certs\") pod \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " Nov 02 02:12:58.421858 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.421718 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istiod-ca-cert\") pod \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " Nov 02 02:12:58.421858 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.421760 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-credential-socket\") pod \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " Nov 02 02:12:58.421858 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.421800 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-data\") pod \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " Nov 02 02:12:58.421858 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.421844 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-podinfo\") pod \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " Nov 02 02:12:58.422126 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.421904 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b500c78-94b3-46ff-a95f-abdc414c5f3b-kube-api-access-j9j4v\" (UniqueName: \"kubernetes.io/projected/7b500c78-94b3-46ff-a95f-abdc414c5f3b-kube-api-access-j9j4v\") pod \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " Nov 02 02:12:58.422126 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.421957 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-token\" (UniqueName: \"kubernetes.io/projected/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-token\") pod \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " Nov 02 02:12:58.422126 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.422082 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-envoy\") pod \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " Nov 02 02:12:58.422126 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.422118 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-workload-socket\") pod \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\" (UID: \"7b500c78-94b3-46ff-a95f-abdc414c5f3b\") " Nov 02 02:12:58.422646 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.422569 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-data" pod "7b500c78-94b3-46ff-a95f-abdc414c5f3b" (UID: "7b500c78-94b3-46ff-a95f-abdc414c5f3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:12:58.422822 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.422790 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istiod-ca-cert" pod "7b500c78-94b3-46ff-a95f-abdc414c5f3b" (UID: "7b500c78-94b3-46ff-a95f-abdc414c5f3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:12:58.423111 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.423074 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-workload-certs" pod "7b500c78-94b3-46ff-a95f-abdc414c5f3b" (UID: "7b500c78-94b3-46ff-a95f-abdc414c5f3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:12:58.424129 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.423859 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-credential-socket" pod "7b500c78-94b3-46ff-a95f-abdc414c5f3b" (UID: "7b500c78-94b3-46ff-a95f-abdc414c5f3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:12:58.424432 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.424400 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-workload-socket" pod "7b500c78-94b3-46ff-a95f-abdc414c5f3b" (UID: "7b500c78-94b3-46ff-a95f-abdc414c5f3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:12:58.426005 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.425974 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-podinfo" pod "7b500c78-94b3-46ff-a95f-abdc414c5f3b" (UID: "7b500c78-94b3-46ff-a95f-abdc414c5f3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 02:12:58.427427 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.427402 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-envoy" pod "7b500c78-94b3-46ff-a95f-abdc414c5f3b" (UID: "7b500c78-94b3-46ff-a95f-abdc414c5f3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:12:58.428078 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.428039 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-token" pod "7b500c78-94b3-46ff-a95f-abdc414c5f3b" (UID: "7b500c78-94b3-46ff-a95f-abdc414c5f3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:12:58.430184 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.430158 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b500c78-94b3-46ff-a95f-abdc414c5f3b-kube-api-access-j9j4v" pod "7b500c78-94b3-46ff-a95f-abdc414c5f3b" (UID: "7b500c78-94b3-46ff-a95f-abdc414c5f3b"). InnerVolumeSpecName "kube-api-access-j9j4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:12:58.459608 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.459588 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 02:12:58.475589 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.475569 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:12:58.523692 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.523659 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-kube-api-access-5rxfv\" (UniqueName: \"kubernetes.io/projected/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-kube-api-access-5rxfv\") pod \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " Nov 02 02:12:58.523787 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.523706 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-podinfo\") pod \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " Nov 02 02:12:58.523787 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.523737 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-envoy\") pod \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " Nov 02 02:12:58.523787 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.523760 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istiod-ca-cert\") pod \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " Nov 02 02:12:58.523787 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.523778 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-kube-api-access-scb6h\" (UniqueName: \"kubernetes.io/projected/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-kube-api-access-scb6h\") pod \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " Nov 02 02:12:58.524006 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.523795 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-credential-socket\") pod \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " Nov 02 02:12:58.524006 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.523815 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-token\" (UniqueName: \"kubernetes.io/projected/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-token\") pod \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " Nov 02 02:12:58.524006 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.523843 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-podinfo\") pod \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " Nov 02 02:12:58.524006 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.523873 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-workload-certs\") pod \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " Nov 02 02:12:58.524006 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.523903 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-token\" (UniqueName: \"kubernetes.io/projected/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-token\") pod \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " Nov 02 02:12:58.524006 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.523951 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-data\") pod \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " Nov 02 02:12:58.524006 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.523984 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istiod-ca-cert\") pod \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " Nov 02 02:12:58.524374 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.524028 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-data\") pod \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " Nov 02 02:12:58.524374 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.524060 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-podinfo\") pod \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " Nov 02 02:12:58.524374 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.524112 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-workload-socket\") pod \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " Nov 02 02:12:58.524374 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.524144 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-credential-socket\") pod \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " Nov 02 02:12:58.524374 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.524188 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-envoy\") pod \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " Nov 02 02:12:58.524374 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.524224 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-data\") pod \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " Nov 02 02:12:58.524374 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.524249 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-credential-socket" pod "766dfcbb-afda-4f73-8b00-8c53e2fb45d5" (UID: "766dfcbb-afda-4f73-8b00-8c53e2fb45d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:12:58.524374 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.524274 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-token\" (UniqueName: \"kubernetes.io/projected/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-token\") pod \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " Nov 02 02:12:58.524374 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.524318 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-workload-certs\") pod \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " Nov 02 02:12:58.524374 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.524360 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-workload-socket\") pod \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\" (UID: \"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc\") " Nov 02 02:12:58.524953 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.524396 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-workload-socket\") pod \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " Nov 02 02:12:58.524953 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.524442 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-workload-certs\") pod \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " Nov 02 02:12:58.524953 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.524471 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istiod-ca-cert\") pod \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " Nov 02 02:12:58.524953 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.524472 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-data" pod "e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" (UID: "e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:12:58.524953 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.524665 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-workload-certs" pod "da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" (UID: "da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:12:58.525227 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.525107 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istiod-ca-cert" pod "da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" (UID: "da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:12:58.526189 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.525360 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-credential-socket" pod "e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" (UID: "e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:12:58.526189 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.525474 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-workload-certs" pod "e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" (UID: "e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:12:58.526189 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.525717 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-workload-certs" pod "766dfcbb-afda-4f73-8b00-8c53e2fb45d5" (UID: "766dfcbb-afda-4f73-8b00-8c53e2fb45d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:12:58.526189 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.525718 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istiod-ca-cert" pod "e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" (UID: "e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:12:58.526189 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.525740 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-workload-socket" pod "da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" (UID: "da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:12:58.526189 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.525764 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-workload-socket" pod "e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" (UID: "e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:12:58.526189 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.525781 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-data" pod "da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" (UID: "da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:12:58.526189 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.525838 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-data" pod "766dfcbb-afda-4f73-8b00-8c53e2fb45d5" (UID: "766dfcbb-afda-4f73-8b00-8c53e2fb45d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:12:58.526189 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.525852 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-workload-socket" pod "766dfcbb-afda-4f73-8b00-8c53e2fb45d5" (UID: "766dfcbb-afda-4f73-8b00-8c53e2fb45d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:12:58.526189 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.525887 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-credential-socket\") pod \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\" (UID: \"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15\") " Nov 02 02:12:58.526189 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.525934 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-kube-api-access-s5p96\" (UniqueName: \"kubernetes.io/projected/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-kube-api-access-s5p96\") pod \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " Nov 02 02:12:58.526189 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.525974 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-envoy\") pod \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\" (UID: \"766dfcbb-afda-4f73-8b00-8c53e2fb45d5\") " Nov 02 02:12:58.526189 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526085 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-credential-socket" pod "da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" (UID: "da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:12:58.526189 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526152 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istiod-ca-cert" pod "766dfcbb-afda-4f73-8b00-8c53e2fb45d5" (UID: "766dfcbb-afda-4f73-8b00-8c53e2fb45d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526445 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526467 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526482 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526497 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526511 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526524 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526537 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526550 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526564 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526578 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526591 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526605 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526620 5044 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istiod-ca-cert\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526636 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526651 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526682 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526696 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526709 5044 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-credential-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526724 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9j4v\" (UniqueName: \"kubernetes.io/projected/7b500c78-94b3-46ff-a95f-abdc414c5f3b-kube-api-access-j9j4v\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526740 5044 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-data\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526756 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b500c78-94b3-46ff-a95f-abdc414c5f3b-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526770 5044 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-workload-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526785 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.526799 5044 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-workload-socket\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.528853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.528294 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-kube-api-access-5rxfv" pod "e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" (UID: "e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc"). InnerVolumeSpecName "kube-api-access-5rxfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:12:58.531297 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.528899 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-podinfo" pod "e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" (UID: "e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 02:12:58.531297 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.529000 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-kube-api-access-scb6h" pod "da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" (UID: "da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15"). InnerVolumeSpecName "kube-api-access-scb6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:12:58.531297 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.529723 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-podinfo" pod "da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" (UID: "da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 02:12:58.531297 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.529662 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-envoy" pod "766dfcbb-afda-4f73-8b00-8c53e2fb45d5" (UID: "766dfcbb-afda-4f73-8b00-8c53e2fb45d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:12:58.531297 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.529949 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-token" pod "da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" (UID: "da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:12:58.531297 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.530122 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-envoy" pod "da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" (UID: "da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:12:58.531297 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.530302 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-envoy" pod "e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" (UID: "e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:12:58.531297 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.530563 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-token" pod "e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" (UID: "e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:12:58.531717 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.531527 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-kube-api-access-s5p96" pod "766dfcbb-afda-4f73-8b00-8c53e2fb45d5" (UID: "766dfcbb-afda-4f73-8b00-8c53e2fb45d5"). InnerVolumeSpecName "kube-api-access-s5p96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:12:58.531717 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.531623 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-token" pod "766dfcbb-afda-4f73-8b00-8c53e2fb45d5" (UID: "766dfcbb-afda-4f73-8b00-8c53e2fb45d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:12:58.533486 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.533460 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-podinfo" pod "766dfcbb-afda-4f73-8b00-8c53e2fb45d5" (UID: "766dfcbb-afda-4f73-8b00-8c53e2fb45d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 02:12:58.627407 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.627334 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.627407 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.627361 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.627407 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.627371 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5p96\" (UniqueName: \"kubernetes.io/projected/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-kube-api-access-s5p96\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.627407 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.627382 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.627407 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.627392 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rxfv\" (UniqueName: \"kubernetes.io/projected/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-kube-api-access-5rxfv\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.627407 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.627401 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.627407 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.627408 5044 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-envoy\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.627739 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.627420 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scb6h\" (UniqueName: \"kubernetes.io/projected/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-kube-api-access-scb6h\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.627739 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.627429 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.627739 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.627440 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.627739 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.627450 5044 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc-istio-token\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.627739 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.627458 5044 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/766dfcbb-afda-4f73-8b00-8c53e2fb45d5-istio-podinfo\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:12:58.791411 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.791373 5044 generic.go:356] "Generic (PLEG): container finished" podID="7b500c78-94b3-46ff-a95f-abdc414c5f3b" containerID="4c692abd1cdf223ce8890af6517f2cb3e27de5a6567954233680a8b8d1d65999" exitCode=0 Nov 02 02:12:58.791579 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.791434 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" Nov 02 02:12:58.791579 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.791452 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" event={"ID":"7b500c78-94b3-46ff-a95f-abdc414c5f3b","Type":"ContainerDied","Data":"4c692abd1cdf223ce8890af6517f2cb3e27de5a6567954233680a8b8d1d65999"} Nov 02 02:12:58.791579 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.791514 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf" event={"ID":"7b500c78-94b3-46ff-a95f-abdc414c5f3b","Type":"ContainerDied","Data":"debf0af021d9bc943c8ed5a9228891a2a52771c0f3cec4a1650b931d7dd38410"} Nov 02 02:12:58.791579 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.791544 5044 scope.go:122] "RemoveContainer" containerID="4c692abd1cdf223ce8890af6517f2cb3e27de5a6567954233680a8b8d1d65999" Nov 02 02:12:58.793546 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.793520 5044 generic.go:356] "Generic (PLEG): container finished" podID="e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" containerID="50b67d2dd2612e3b9b0c3f71d45475217ef8439fe8266bb5f4645e15cd2b6df2" exitCode=0 Nov 02 02:12:58.793815 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.793573 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" Nov 02 02:12:58.794108 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.793593 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" event={"ID":"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc","Type":"ContainerDied","Data":"50b67d2dd2612e3b9b0c3f71d45475217ef8439fe8266bb5f4645e15cd2b6df2"} Nov 02 02:12:58.794108 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.794066 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj" event={"ID":"e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc","Type":"ContainerDied","Data":"983cbf2b6a1689b34c47602941870d8104b449d5b5c2b767347938a1e474042b"} Nov 02 02:12:58.796153 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.796115 5044 generic.go:356] "Generic (PLEG): container finished" podID="da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" containerID="5d246185e91e96a8a1e5dddd099a6b961167a701338521d5548af550570d7ae5" exitCode=0 Nov 02 02:12:58.796256 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.796218 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" event={"ID":"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15","Type":"ContainerDied","Data":"5d246185e91e96a8a1e5dddd099a6b961167a701338521d5548af550570d7ae5"} Nov 02 02:12:58.796328 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.796260 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" event={"ID":"da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15","Type":"ContainerDied","Data":"406f61a772a2debc2b2fae0a1808546ad509ff273d7570e4de055f2a05439298"} Nov 02 02:12:58.796387 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.796338 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk" Nov 02 02:12:58.804442 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.804424 5044 scope.go:122] "RemoveContainer" containerID="4c692abd1cdf223ce8890af6517f2cb3e27de5a6567954233680a8b8d1d65999" Nov 02 02:12:58.805098 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:12:58.805049 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c692abd1cdf223ce8890af6517f2cb3e27de5a6567954233680a8b8d1d65999\": container with ID starting with 4c692abd1cdf223ce8890af6517f2cb3e27de5a6567954233680a8b8d1d65999 not found: ID does not exist" containerID="4c692abd1cdf223ce8890af6517f2cb3e27de5a6567954233680a8b8d1d65999" Nov 02 02:12:58.805098 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.805084 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c692abd1cdf223ce8890af6517f2cb3e27de5a6567954233680a8b8d1d65999"} err="failed to get container status \"4c692abd1cdf223ce8890af6517f2cb3e27de5a6567954233680a8b8d1d65999\": rpc error: code = NotFound desc = could not find container \"4c692abd1cdf223ce8890af6517f2cb3e27de5a6567954233680a8b8d1d65999\": container with ID starting with 4c692abd1cdf223ce8890af6517f2cb3e27de5a6567954233680a8b8d1d65999 not found: ID does not exist" Nov 02 02:12:58.805236 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.805107 5044 scope.go:122] "RemoveContainer" containerID="50b67d2dd2612e3b9b0c3f71d45475217ef8439fe8266bb5f4645e15cd2b6df2" Nov 02 02:12:58.806377 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.806350 5044 generic.go:356] "Generic (PLEG): container finished" podID="766dfcbb-afda-4f73-8b00-8c53e2fb45d5" containerID="fb0067c9c9529e478e01367b5810cc450623aaaac89f0e769248a9c38dcd4fd7" exitCode=0 Nov 02 02:12:58.806483 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.806457 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" event={"ID":"766dfcbb-afda-4f73-8b00-8c53e2fb45d5","Type":"ContainerDied","Data":"fb0067c9c9529e478e01367b5810cc450623aaaac89f0e769248a9c38dcd4fd7"} Nov 02 02:12:58.806549 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.806500 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" event={"ID":"766dfcbb-afda-4f73-8b00-8c53e2fb45d5","Type":"ContainerDied","Data":"31445b5342c113648255090c5ea98f3552b51ad8a4e78974a035346780b19011"} Nov 02 02:12:58.806681 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.806646 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b" Nov 02 02:12:58.819441 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.819426 5044 scope.go:122] "RemoveContainer" containerID="50b67d2dd2612e3b9b0c3f71d45475217ef8439fe8266bb5f4645e15cd2b6df2" Nov 02 02:12:58.819735 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:12:58.819713 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b67d2dd2612e3b9b0c3f71d45475217ef8439fe8266bb5f4645e15cd2b6df2\": container with ID starting with 50b67d2dd2612e3b9b0c3f71d45475217ef8439fe8266bb5f4645e15cd2b6df2 not found: ID does not exist" containerID="50b67d2dd2612e3b9b0c3f71d45475217ef8439fe8266bb5f4645e15cd2b6df2" Nov 02 02:12:58.819829 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.819747 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b67d2dd2612e3b9b0c3f71d45475217ef8439fe8266bb5f4645e15cd2b6df2"} err="failed to get container status \"50b67d2dd2612e3b9b0c3f71d45475217ef8439fe8266bb5f4645e15cd2b6df2\": rpc error: code = NotFound desc = could not find container \"50b67d2dd2612e3b9b0c3f71d45475217ef8439fe8266bb5f4645e15cd2b6df2\": container with ID starting with 50b67d2dd2612e3b9b0c3f71d45475217ef8439fe8266bb5f4645e15cd2b6df2 not found: ID does not exist" Nov 02 02:12:58.819829 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.819764 5044 scope.go:122] "RemoveContainer" containerID="5d246185e91e96a8a1e5dddd099a6b961167a701338521d5548af550570d7ae5" Nov 02 02:12:58.828601 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.828582 5044 scope.go:122] "RemoveContainer" containerID="5d246185e91e96a8a1e5dddd099a6b961167a701338521d5548af550570d7ae5" Nov 02 02:12:58.829041 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:12:58.829015 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d246185e91e96a8a1e5dddd099a6b961167a701338521d5548af550570d7ae5\": container with ID starting with 5d246185e91e96a8a1e5dddd099a6b961167a701338521d5548af550570d7ae5 not found: ID does not exist" containerID="5d246185e91e96a8a1e5dddd099a6b961167a701338521d5548af550570d7ae5" Nov 02 02:12:58.829110 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.829051 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d246185e91e96a8a1e5dddd099a6b961167a701338521d5548af550570d7ae5"} err="failed to get container status \"5d246185e91e96a8a1e5dddd099a6b961167a701338521d5548af550570d7ae5\": rpc error: code = NotFound desc = could not find container \"5d246185e91e96a8a1e5dddd099a6b961167a701338521d5548af550570d7ae5\": container with ID starting with 5d246185e91e96a8a1e5dddd099a6b961167a701338521d5548af550570d7ae5 not found: ID does not exist" Nov 02 02:12:58.829110 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.829076 5044 scope.go:122] "RemoveContainer" containerID="fb0067c9c9529e478e01367b5810cc450623aaaac89f0e769248a9c38dcd4fd7" Nov 02 02:12:58.839535 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.839512 5044 scope.go:122] "RemoveContainer" containerID="fb0067c9c9529e478e01367b5810cc450623aaaac89f0e769248a9c38dcd4fd7" Nov 02 02:12:58.839823 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:12:58.839800 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0067c9c9529e478e01367b5810cc450623aaaac89f0e769248a9c38dcd4fd7\": container with ID starting with fb0067c9c9529e478e01367b5810cc450623aaaac89f0e769248a9c38dcd4fd7 not found: ID does not exist" containerID="fb0067c9c9529e478e01367b5810cc450623aaaac89f0e769248a9c38dcd4fd7" Nov 02 02:12:58.839906 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:58.839830 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0067c9c9529e478e01367b5810cc450623aaaac89f0e769248a9c38dcd4fd7"} err="failed to get container status \"fb0067c9c9529e478e01367b5810cc450623aaaac89f0e769248a9c38dcd4fd7\": rpc error: code = NotFound desc = could not find container \"fb0067c9c9529e478e01367b5810cc450623aaaac89f0e769248a9c38dcd4fd7\": container with ID starting with fb0067c9c9529e478e01367b5810cc450623aaaac89f0e769248a9c38dcd4fd7 not found: ID does not exist" Nov 02 02:12:59.018988 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:59.018950 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b"] Nov 02 02:12:59.023550 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:59.023526 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-qwp4b"] Nov 02 02:12:59.136882 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:59.136848 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf"] Nov 02 02:12:59.141704 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:59.141664 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rhjxf"] Nov 02 02:12:59.192565 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:59.192537 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:59.253761 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:59.253728 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk"] Nov 02 02:12:59.258766 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:59.258744 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-pjltk"] Nov 02 02:12:59.341305 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:59.341201 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:12:59.373251 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:59.373221 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj"] Nov 02 02:12:59.378565 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:59.378536 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9fjhj"] Nov 02 02:12:59.451453 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:59.451430 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="766dfcbb-afda-4f73-8b00-8c53e2fb45d5" path="/var/lib/kubelet/pods/766dfcbb-afda-4f73-8b00-8c53e2fb45d5/volumes" Nov 02 02:12:59.451884 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:59.451870 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b500c78-94b3-46ff-a95f-abdc414c5f3b" path="/var/lib/kubelet/pods/7b500c78-94b3-46ff-a95f-abdc414c5f3b/volumes" Nov 02 02:12:59.452264 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:59.452253 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" path="/var/lib/kubelet/pods/da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15/volumes" Nov 02 02:12:59.453126 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:12:59.453108 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" path="/var/lib/kubelet/pods/e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc/volumes" Nov 02 02:13:01.782311 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:01.782266 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:06.586349 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:06.586311 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5"] Nov 02 02:13:06.594007 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:06.593978 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:06.841704 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:06.841579 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:06.985643 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:06.985613 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:10.657431 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:10.657396 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92"] Nov 02 02:13:10.657866 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:10.657602 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" podUID="380ec44f-2ae2-4c12-9f95-5c6e424de57c" containerName="maas-api" containerID="cri-o://2bbb20a2873a92c54011d6158991dbc24b77643c63f05c1cee82326e097f3b1e" gracePeriod=30 Nov 02 02:13:10.760535 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:10.760505 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:10.883996 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:10.883963 5044 generic.go:356] "Generic (PLEG): container finished" podID="380ec44f-2ae2-4c12-9f95-5c6e424de57c" containerID="2bbb20a2873a92c54011d6158991dbc24b77643c63f05c1cee82326e097f3b1e" exitCode=0 Nov 02 02:13:10.884162 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:10.884031 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" event={"ID":"380ec44f-2ae2-4c12-9f95-5c6e424de57c","Type":"ContainerDied","Data":"2bbb20a2873a92c54011d6158991dbc24b77643c63f05c1cee82326e097f3b1e"} Nov 02 02:13:10.928618 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:10.928594 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" Nov 02 02:13:11.050978 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:11.050280 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/380ec44f-2ae2-4c12-9f95-5c6e424de57c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/380ec44f-2ae2-4c12-9f95-5c6e424de57c-metrics-certs\") pod \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\" (UID: \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\") " Nov 02 02:13:11.050978 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:11.050402 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/380ec44f-2ae2-4c12-9f95-5c6e424de57c-kube-api-access-gctx8\" (UniqueName: \"kubernetes.io/projected/380ec44f-2ae2-4c12-9f95-5c6e424de57c-kube-api-access-gctx8\") pod \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\" (UID: \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\") " Nov 02 02:13:11.050978 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:11.050436 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/380ec44f-2ae2-4c12-9f95-5c6e424de57c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380ec44f-2ae2-4c12-9f95-5c6e424de57c-maas-api-tls\") pod \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\" (UID: \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\") " Nov 02 02:13:11.050978 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:11.050488 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/380ec44f-2ae2-4c12-9f95-5c6e424de57c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380ec44f-2ae2-4c12-9f95-5c6e424de57c-openshift-service-ca\") pod \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\" (UID: \"380ec44f-2ae2-4c12-9f95-5c6e424de57c\") " Nov 02 02:13:11.050978 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:11.050965 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380ec44f-2ae2-4c12-9f95-5c6e424de57c-openshift-service-ca" pod "380ec44f-2ae2-4c12-9f95-5c6e424de57c" (UID: "380ec44f-2ae2-4c12-9f95-5c6e424de57c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:13:11.051720 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:11.051665 5044 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/380ec44f-2ae2-4c12-9f95-5c6e424de57c-openshift-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:13:11.052971 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:11.052942 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/380ec44f-2ae2-4c12-9f95-5c6e424de57c-metrics-certs" pod "380ec44f-2ae2-4c12-9f95-5c6e424de57c" (UID: "380ec44f-2ae2-4c12-9f95-5c6e424de57c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:13:11.053179 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:11.053149 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/380ec44f-2ae2-4c12-9f95-5c6e424de57c-maas-api-tls" pod "380ec44f-2ae2-4c12-9f95-5c6e424de57c" (UID: "380ec44f-2ae2-4c12-9f95-5c6e424de57c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:13:11.053818 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:11.053791 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380ec44f-2ae2-4c12-9f95-5c6e424de57c-kube-api-access-gctx8" pod "380ec44f-2ae2-4c12-9f95-5c6e424de57c" (UID: "380ec44f-2ae2-4c12-9f95-5c6e424de57c"). InnerVolumeSpecName "kube-api-access-gctx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:13:11.152324 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:11.152292 5044 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/380ec44f-2ae2-4c12-9f95-5c6e424de57c-metrics-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:13:11.152324 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:11.152320 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gctx8\" (UniqueName: \"kubernetes.io/projected/380ec44f-2ae2-4c12-9f95-5c6e424de57c-kube-api-access-gctx8\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:13:11.152509 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:11.152335 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/380ec44f-2ae2-4c12-9f95-5c6e424de57c-maas-api-tls\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:13:11.587476 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:11.587431 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5" podUID="4e55a287-c53e-49bf-88ad-21e4ede955a6" containerName="payload-pre-processing" containerID="cri-o://5af640670d569aa7c6499c939c8ad4a03923a7409ce127979df43993eb423eac" gracePeriod=25 Nov 02 02:13:11.890587 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:11.890552 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" event={"ID":"380ec44f-2ae2-4c12-9f95-5c6e424de57c","Type":"ContainerDied","Data":"f1aeca22a67a65e3aa80c497e12c21835e51d6abb5caa3cc0f2f3c8d2b350e68"} Nov 02 02:13:11.890587 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:11.890568 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92" Nov 02 02:13:11.891126 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:11.890619 5044 scope.go:122] "RemoveContainer" containerID="2bbb20a2873a92c54011d6158991dbc24b77643c63f05c1cee82326e097f3b1e" Nov 02 02:13:12.059092 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:12.059061 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92"] Nov 02 02:13:12.110329 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:12.110301 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-361fcc-859c9bf46c-vzm92"] Nov 02 02:13:12.257091 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:12.257011 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:12.477266 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:12.477234 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:12.616878 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:12.616851 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:13.170099 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:13.170064 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:13.450860 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:13.450786 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="380ec44f-2ae2-4c12-9f95-5c6e424de57c" path="/var/lib/kubelet/pods/380ec44f-2ae2-4c12-9f95-5c6e424de57c/volumes" Nov 02 02:13:16.305635 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:16.305597 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5" podUID="4e55a287-c53e-49bf-88ad-21e4ede955a6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 02:13:17.489094 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:17.489030 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:19.241457 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:19.241422 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:19.384350 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:19.384307 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:19.575532 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:19.575502 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:22.558998 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:22.558966 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:24.176584 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:13:24.176546 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:13:26.305963 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:26.305925 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5" podUID="4e55a287-c53e-49bf-88ad-21e4ede955a6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 02:13:27.568420 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:27.568374 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:31.631589 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:31.631557 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd"] Nov 02 02:13:31.631950 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:31.631797 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" podUID="4052c546-2071-483f-878f-290e59139928" containerName="maas-api" containerID="cri-o://9b48f02f3793dc25d3230879f20b44f4cb38942dd7d3b965749c66e79b51bc12" gracePeriod=30 Nov 02 02:13:31.729473 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:31.729443 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:31.914583 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:31.914556 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5" Nov 02 02:13:31.938151 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:31.938128 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e55a287-c53e-49bf-88ad-21e4ede955a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e55a287-c53e-49bf-88ad-21e4ede955a6-plugins-config-volume\") pod \"4e55a287-c53e-49bf-88ad-21e4ede955a6\" (UID: \"4e55a287-c53e-49bf-88ad-21e4ede955a6\") " Nov 02 02:13:31.938289 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:31.938271 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e55a287-c53e-49bf-88ad-21e4ede955a6-kube-api-access-v92vm\" (UniqueName: \"kubernetes.io/projected/4e55a287-c53e-49bf-88ad-21e4ede955a6-kube-api-access-v92vm\") pod \"4e55a287-c53e-49bf-88ad-21e4ede955a6\" (UID: \"4e55a287-c53e-49bf-88ad-21e4ede955a6\") " Nov 02 02:13:31.938512 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:31.938490 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e55a287-c53e-49bf-88ad-21e4ede955a6-plugins-config-volume" pod "4e55a287-c53e-49bf-88ad-21e4ede955a6" (UID: "4e55a287-c53e-49bf-88ad-21e4ede955a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:13:31.940419 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:31.940399 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e55a287-c53e-49bf-88ad-21e4ede955a6-kube-api-access-v92vm" pod "4e55a287-c53e-49bf-88ad-21e4ede955a6" (UID: "4e55a287-c53e-49bf-88ad-21e4ede955a6"). InnerVolumeSpecName "kube-api-access-v92vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:13:31.943261 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:31.943245 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" Nov 02 02:13:32.003633 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.003595 5044 generic.go:356] "Generic (PLEG): container finished" podID="4052c546-2071-483f-878f-290e59139928" containerID="9b48f02f3793dc25d3230879f20b44f4cb38942dd7d3b965749c66e79b51bc12" exitCode=0 Nov 02 02:13:32.003838 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.003777 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" event={"ID":"4052c546-2071-483f-878f-290e59139928","Type":"ContainerDied","Data":"9b48f02f3793dc25d3230879f20b44f4cb38942dd7d3b965749c66e79b51bc12"} Nov 02 02:13:32.003838 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.003792 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" Nov 02 02:13:32.003838 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.003835 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd" event={"ID":"4052c546-2071-483f-878f-290e59139928","Type":"ContainerDied","Data":"f7989f02f1a03fd4d4fd5d8d67c54223213cf11a8e6e6b0240be244e30b5e629"} Nov 02 02:13:32.004062 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.003863 5044 scope.go:122] "RemoveContainer" containerID="9b48f02f3793dc25d3230879f20b44f4cb38942dd7d3b965749c66e79b51bc12" Nov 02 02:13:32.006884 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.006854 5044 generic.go:356] "Generic (PLEG): container finished" podID="4e55a287-c53e-49bf-88ad-21e4ede955a6" containerID="5af640670d569aa7c6499c939c8ad4a03923a7409ce127979df43993eb423eac" exitCode=0 Nov 02 02:13:32.006987 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.006938 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5" event={"ID":"4e55a287-c53e-49bf-88ad-21e4ede955a6","Type":"ContainerDied","Data":"5af640670d569aa7c6499c939c8ad4a03923a7409ce127979df43993eb423eac"} Nov 02 02:13:32.007048 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.006988 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5" event={"ID":"4e55a287-c53e-49bf-88ad-21e4ede955a6","Type":"ContainerDied","Data":"0fd6a497facac5125c0c01317f7af41cc223353cf9b35f818a219f99d1024bd2"} Nov 02 02:13:32.007048 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.007040 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5" Nov 02 02:13:32.015147 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.015130 5044 scope.go:122] "RemoveContainer" containerID="9b48f02f3793dc25d3230879f20b44f4cb38942dd7d3b965749c66e79b51bc12" Nov 02 02:13:32.015452 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:13:32.015395 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b48f02f3793dc25d3230879f20b44f4cb38942dd7d3b965749c66e79b51bc12\": container with ID starting with 9b48f02f3793dc25d3230879f20b44f4cb38942dd7d3b965749c66e79b51bc12 not found: ID does not exist" containerID="9b48f02f3793dc25d3230879f20b44f4cb38942dd7d3b965749c66e79b51bc12" Nov 02 02:13:32.015452 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.015430 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b48f02f3793dc25d3230879f20b44f4cb38942dd7d3b965749c66e79b51bc12"} err="failed to get container status \"9b48f02f3793dc25d3230879f20b44f4cb38942dd7d3b965749c66e79b51bc12\": rpc error: code = NotFound desc = could not find container \"9b48f02f3793dc25d3230879f20b44f4cb38942dd7d3b965749c66e79b51bc12\": container with ID starting with 9b48f02f3793dc25d3230879f20b44f4cb38942dd7d3b965749c66e79b51bc12 not found: ID does not exist" Nov 02 02:13:32.015580 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.015454 5044 scope.go:122] "RemoveContainer" containerID="5af640670d569aa7c6499c939c8ad4a03923a7409ce127979df43993eb423eac" Nov 02 02:13:32.023502 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.023485 5044 scope.go:122] "RemoveContainer" containerID="5af640670d569aa7c6499c939c8ad4a03923a7409ce127979df43993eb423eac" Nov 02 02:13:32.023805 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:13:32.023786 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af640670d569aa7c6499c939c8ad4a03923a7409ce127979df43993eb423eac\": container with ID starting with 5af640670d569aa7c6499c939c8ad4a03923a7409ce127979df43993eb423eac not found: ID does not exist" containerID="5af640670d569aa7c6499c939c8ad4a03923a7409ce127979df43993eb423eac" Nov 02 02:13:32.023874 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.023811 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af640670d569aa7c6499c939c8ad4a03923a7409ce127979df43993eb423eac"} err="failed to get container status \"5af640670d569aa7c6499c939c8ad4a03923a7409ce127979df43993eb423eac\": rpc error: code = NotFound desc = could not find container \"5af640670d569aa7c6499c939c8ad4a03923a7409ce127979df43993eb423eac\": container with ID starting with 5af640670d569aa7c6499c939c8ad4a03923a7409ce127979df43993eb423eac not found: ID does not exist" Nov 02 02:13:32.039352 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.039319 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4052c546-2071-483f-878f-290e59139928-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4052c546-2071-483f-878f-290e59139928-maas-api-tls\") pod \"4052c546-2071-483f-878f-290e59139928\" (UID: \"4052c546-2071-483f-878f-290e59139928\") " Nov 02 02:13:32.039479 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.039383 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4052c546-2071-483f-878f-290e59139928-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4052c546-2071-483f-878f-290e59139928-metrics-certs\") pod \"4052c546-2071-483f-878f-290e59139928\" (UID: \"4052c546-2071-483f-878f-290e59139928\") " Nov 02 02:13:32.039479 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.039436 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4052c546-2071-483f-878f-290e59139928-kube-api-access-8bbh8\" (UniqueName: \"kubernetes.io/projected/4052c546-2071-483f-878f-290e59139928-kube-api-access-8bbh8\") pod \"4052c546-2071-483f-878f-290e59139928\" (UID: \"4052c546-2071-483f-878f-290e59139928\") " Nov 02 02:13:32.039597 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.039532 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4052c546-2071-483f-878f-290e59139928-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4052c546-2071-483f-878f-290e59139928-openshift-service-ca\") pod \"4052c546-2071-483f-878f-290e59139928\" (UID: \"4052c546-2071-483f-878f-290e59139928\") " Nov 02 02:13:32.040263 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.040240 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v92vm\" (UniqueName: \"kubernetes.io/projected/4e55a287-c53e-49bf-88ad-21e4ede955a6-kube-api-access-v92vm\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:13:32.040407 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.040265 5044 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e55a287-c53e-49bf-88ad-21e4ede955a6-plugins-config-volume\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:13:32.040813 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.040786 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4052c546-2071-483f-878f-290e59139928-openshift-service-ca" pod "4052c546-2071-483f-878f-290e59139928" (UID: "4052c546-2071-483f-878f-290e59139928"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:13:32.043101 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.043072 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4052c546-2071-483f-878f-290e59139928-metrics-certs" pod "4052c546-2071-483f-878f-290e59139928" (UID: "4052c546-2071-483f-878f-290e59139928"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:13:32.043193 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.043141 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4052c546-2071-483f-878f-290e59139928-maas-api-tls" pod "4052c546-2071-483f-878f-290e59139928" (UID: "4052c546-2071-483f-878f-290e59139928"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:13:32.043267 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.043198 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4052c546-2071-483f-878f-290e59139928-kube-api-access-8bbh8" pod "4052c546-2071-483f-878f-290e59139928" (UID: "4052c546-2071-483f-878f-290e59139928"). InnerVolumeSpecName "kube-api-access-8bbh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:13:32.097198 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.097174 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:32.141067 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.140999 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bbh8\" (UniqueName: \"kubernetes.io/projected/4052c546-2071-483f-878f-290e59139928-kube-api-access-8bbh8\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:13:32.141067 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.141026 5044 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4052c546-2071-483f-878f-290e59139928-openshift-service-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:13:32.141067 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.141043 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4052c546-2071-483f-878f-290e59139928-maas-api-tls\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:13:32.141067 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.141065 5044 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4052c546-2071-483f-878f-290e59139928-metrics-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:13:32.247307 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.247268 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:32.277841 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.277790 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5"] Nov 02 02:13:32.280715 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.280665 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-361fcc-5f595864d7-t7ff5"] Nov 02 02:13:32.429952 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.429872 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd"] Nov 02 02:13:32.435500 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:32.435475 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-257466-fb7b9bb7f-k57wd"] Nov 02 02:13:33.451869 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:33.451837 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4052c546-2071-483f-878f-290e59139928" path="/var/lib/kubelet/pods/4052c546-2071-483f-878f-290e59139928/volumes" Nov 02 02:13:33.452381 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:33.452360 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e55a287-c53e-49bf-88ad-21e4ede955a6" path="/var/lib/kubelet/pods/4e55a287-c53e-49bf-88ad-21e4ede955a6/volumes" Nov 02 02:13:33.476677 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:33.476652 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:36.049584 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:36.049545 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:36.334541 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:36.334455 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:36.984187 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:36.984151 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:38.424286 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:38.424247 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:43.096395 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:43.096359 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:43.263072 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:43.263038 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:43.413597 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:43.413490 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:43.562049 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:43.562012 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:48.575887 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:48.575846 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:49.938453 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:49.938423 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:53.486513 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:53.486474 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:54.172498 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:13:54.172458 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:13:56.194908 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:56.194874 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:56.340773 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:56.340742 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:13:58.191659 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:13:58.191624 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:02.985375 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:02.985342 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:07.208041 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:07.208003 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:08.720284 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:08.720252 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:08.986722 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:08.986639 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:09.135962 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:09.135929 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:09.640795 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:09.640763 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:14.169799 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:14.169767 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:15.728691 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:15.728632 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:15.878163 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:15.878124 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:16.029052 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:16.029014 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:16.453020 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:16.452940 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:19.294475 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:19.294425 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:21.420462 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:21.420421 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv"] Nov 02 02:14:21.471450 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:21.471414 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7"] Nov 02 02:14:21.540597 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:21.540565 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:24.172320 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:14:24.172278 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:14:24.262550 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:24.262520 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:28.490939 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.490900 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr"] Nov 02 02:14:28.618155 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.618106 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4"] Nov 02 02:14:28.619216 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.619183 5044 cpu_manager.go:414] "containerMap: removing container" podUID="bd419016-7531-4bcc-9f84-e07ee7a5d747" containerName="payload-pre-processing" Nov 02 02:14:28.619397 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.619381 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd419016-7531-4bcc-9f84-e07ee7a5d747" containerName="payload-pre-processing" Nov 02 02:14:28.619493 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.619483 5044 cpu_manager.go:414] "containerMap: removing container" podUID="da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" containerName="istio-proxy" Nov 02 02:14:28.619583 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.619572 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" containerName="istio-proxy" Nov 02 02:14:28.619665 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.619655 5044 cpu_manager.go:414] "containerMap: removing container" podUID="4052c546-2071-483f-878f-290e59139928" containerName="maas-api" Nov 02 02:14:28.619797 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.619783 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4052c546-2071-483f-878f-290e59139928" containerName="maas-api" Nov 02 02:14:28.619896 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.619887 5044 cpu_manager.go:414] "containerMap: removing container" podUID="e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" containerName="istio-proxy" Nov 02 02:14:28.619972 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.619960 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" containerName="istio-proxy" Nov 02 02:14:28.620056 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.620047 5044 cpu_manager.go:414] "containerMap: removing container" podUID="0f4ffc26-b174-4586-aa5d-11b08755848a" containerName="payload-pre-processing" Nov 02 02:14:28.620132 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.620122 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f4ffc26-b174-4586-aa5d-11b08755848a" containerName="payload-pre-processing" Nov 02 02:14:28.620211 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.620202 5044 cpu_manager.go:414] "containerMap: removing container" podUID="2e26e23b-5900-423c-9e14-7122ce904c81" containerName="payload-pre-processing" Nov 02 02:14:28.620278 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.620268 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e26e23b-5900-423c-9e14-7122ce904c81" containerName="payload-pre-processing" Nov 02 02:14:28.620360 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.620350 5044 cpu_manager.go:414] "containerMap: removing container" podUID="85abf3dc-3eea-41cc-bbaf-2d0208adc731" containerName="payload-pre-processing" Nov 02 02:14:28.620431 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.620422 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85abf3dc-3eea-41cc-bbaf-2d0208adc731" containerName="payload-pre-processing" Nov 02 02:14:28.620521 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.620512 5044 cpu_manager.go:414] "containerMap: removing container" podUID="7b500c78-94b3-46ff-a95f-abdc414c5f3b" containerName="istio-proxy" Nov 02 02:14:28.620598 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.620589 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b500c78-94b3-46ff-a95f-abdc414c5f3b" containerName="istio-proxy" Nov 02 02:14:28.620704 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.620693 5044 cpu_manager.go:414] "containerMap: removing container" podUID="6a0186e1-b5ca-4475-b1bd-0003168fa850" containerName="maas-api" Nov 02 02:14:28.620794 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.620784 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a0186e1-b5ca-4475-b1bd-0003168fa850" containerName="maas-api" Nov 02 02:14:28.620873 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.620864 5044 cpu_manager.go:414] "containerMap: removing container" podUID="581a88ec-7e87-4e19-b6d8-093d7f2b2cb7" containerName="payload-processing" Nov 02 02:14:28.620940 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.620930 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="581a88ec-7e87-4e19-b6d8-093d7f2b2cb7" containerName="payload-processing" Nov 02 02:14:28.621017 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.621008 5044 cpu_manager.go:414] "containerMap: removing container" podUID="7812bf12-6286-4de7-88b4-b6d509c9482c" containerName="maas-api" Nov 02 02:14:28.621084 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.621074 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7812bf12-6286-4de7-88b4-b6d509c9482c" containerName="maas-api" Nov 02 02:14:28.621155 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.621147 5044 cpu_manager.go:414] "containerMap: removing container" podUID="e16f31d3-d566-4620-a625-097e8ee9e28b" containerName="istio-proxy" Nov 02 02:14:28.621258 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.621247 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16f31d3-d566-4620-a625-097e8ee9e28b" containerName="istio-proxy" Nov 02 02:14:28.621330 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.621321 5044 cpu_manager.go:414] "containerMap: removing container" podUID="4e55a287-c53e-49bf-88ad-21e4ede955a6" containerName="payload-pre-processing" Nov 02 02:14:28.621404 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.621394 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e55a287-c53e-49bf-88ad-21e4ede955a6" containerName="payload-pre-processing" Nov 02 02:14:28.621478 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.621469 5044 cpu_manager.go:414] "containerMap: removing container" podUID="766dfcbb-afda-4f73-8b00-8c53e2fb45d5" containerName="istio-proxy" Nov 02 02:14:28.621545 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.621536 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="766dfcbb-afda-4f73-8b00-8c53e2fb45d5" containerName="istio-proxy" Nov 02 02:14:28.621630 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.621621 5044 cpu_manager.go:414] "containerMap: removing container" podUID="b1122c44-90ba-41b2-a10a-72e917ae092e" containerName="payload-processing" Nov 02 02:14:28.621719 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.621708 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1122c44-90ba-41b2-a10a-72e917ae092e" containerName="payload-processing" Nov 02 02:14:28.621816 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.621807 5044 cpu_manager.go:414] "containerMap: removing container" podUID="380ec44f-2ae2-4c12-9f95-5c6e424de57c" containerName="maas-api" Nov 02 02:14:28.621890 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.621881 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380ec44f-2ae2-4c12-9f95-5c6e424de57c" containerName="maas-api" Nov 02 02:14:28.621971 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.621961 5044 cpu_manager.go:414] "containerMap: removing container" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" Nov 02 02:14:28.622038 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.622030 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" Nov 02 02:14:28.622117 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.622109 5044 cpu_manager.go:414] "containerMap: removing container" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" Nov 02 02:14:28.622185 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.622176 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" Nov 02 02:14:28.622264 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.622256 5044 cpu_manager.go:414] "containerMap: removing container" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" Nov 02 02:14:28.622343 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.622333 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" Nov 02 02:14:28.622443 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.622433 5044 cpu_manager.go:414] "containerMap: removing container" podUID="652e50af-8a91-43ff-bb49-bbdf781feefd" containerName="istio-proxy" Nov 02 02:14:28.622517 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.622508 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652e50af-8a91-43ff-bb49-bbdf781feefd" containerName="istio-proxy" Nov 02 02:14:28.622594 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.622585 5044 cpu_manager.go:414] "containerMap: removing container" podUID="652e50af-8a91-43ff-bb49-bbdf781feefd" containerName="istio-proxy" Nov 02 02:14:28.622662 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.622652 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652e50af-8a91-43ff-bb49-bbdf781feefd" containerName="istio-proxy" Nov 02 02:14:28.622769 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.622758 5044 cpu_manager.go:414] "containerMap: removing container" podUID="dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8" containerName="payload-pre-processing" Nov 02 02:14:28.622841 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.622830 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8" containerName="payload-pre-processing" Nov 02 02:14:28.622919 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.622911 5044 cpu_manager.go:414] "containerMap: removing container" podUID="9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d" containerName="maas-api" Nov 02 02:14:28.622986 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.622977 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d" containerName="maas-api" Nov 02 02:14:28.623075 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.623066 5044 cpu_manager.go:414] "containerMap: removing container" podUID="d7d1b5a4-50ae-46d5-83f9-252cb2e8734c" containerName="payload-processing" Nov 02 02:14:28.623147 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.623138 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d1b5a4-50ae-46d5-83f9-252cb2e8734c" containerName="payload-processing" Nov 02 02:14:28.623430 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.623415 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ee089e-cf8a-4d98-b6b5-4e1f374c3cbc" containerName="istio-proxy" Nov 02 02:14:28.623532 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.623522 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f4ffc26-b174-4586-aa5d-11b08755848a" containerName="payload-pre-processing" Nov 02 02:14:28.623616 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.623607 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e26e23b-5900-423c-9e14-7122ce904c81" containerName="payload-pre-processing" Nov 02 02:14:28.623734 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.623723 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="85abf3dc-3eea-41cc-bbaf-2d0208adc731" containerName="payload-pre-processing" Nov 02 02:14:28.623815 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.623805 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b500c78-94b3-46ff-a95f-abdc414c5f3b" containerName="istio-proxy" Nov 02 02:14:28.623890 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.623881 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a0186e1-b5ca-4475-b1bd-0003168fa850" containerName="maas-api" Nov 02 02:14:28.623961 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.623953 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="581a88ec-7e87-4e19-b6d8-093d7f2b2cb7" containerName="payload-processing" Nov 02 02:14:28.624027 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.624018 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="7812bf12-6286-4de7-88b4-b6d509c9482c" containerName="maas-api" Nov 02 02:14:28.624091 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.624082 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="e16f31d3-d566-4620-a625-097e8ee9e28b" containerName="istio-proxy" Nov 02 02:14:28.624162 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.624153 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e55a287-c53e-49bf-88ad-21e4ede955a6" containerName="payload-pre-processing" Nov 02 02:14:28.624235 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.624225 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="766dfcbb-afda-4f73-8b00-8c53e2fb45d5" containerName="istio-proxy" Nov 02 02:14:28.624314 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.624305 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1122c44-90ba-41b2-a10a-72e917ae092e" containerName="payload-processing" Nov 02 02:14:28.624390 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.624382 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="380ec44f-2ae2-4c12-9f95-5c6e424de57c" containerName="maas-api" Nov 02 02:14:28.624454 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.624445 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd6f3e6-f1f2-4f85-85a4-a2378ddd2cbc" containerName="main" Nov 02 02:14:28.624528 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.624519 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae3a465-c735-401b-a679-0a6141ca6bb9" containerName="main" Nov 02 02:14:28.624595 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.624586 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="39920d19-93e7-46ad-a3a4-90a51b766ab3" containerName="main" Nov 02 02:14:28.624707 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.624696 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="652e50af-8a91-43ff-bb49-bbdf781feefd" containerName="istio-proxy" Nov 02 02:14:28.624852 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.624842 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae5f42f-91d9-49f6-8f7f-84dbffc0f1a8" containerName="payload-pre-processing" Nov 02 02:14:28.624933 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.624925 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ea7ab8f-45cb-4ce3-af7e-896e2e049f0d" containerName="maas-api" Nov 02 02:14:28.625005 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.624996 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d1b5a4-50ae-46d5-83f9-252cb2e8734c" containerName="payload-processing" Nov 02 02:14:28.625089 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.625080 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd419016-7531-4bcc-9f84-e07ee7a5d747" containerName="payload-pre-processing" Nov 02 02:14:28.625169 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.625161 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="da89ef7a-b9e2-4ecf-bcf5-dba89ba3dd15" containerName="istio-proxy" Nov 02 02:14:28.625238 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.625230 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="4052c546-2071-483f-878f-290e59139928" containerName="maas-api" Nov 02 02:14:28.629717 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.629691 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4" Nov 02 02:14:28.649203 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.649178 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4"] Nov 02 02:14:28.716895 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.716860 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 02:14:28.716895 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.716866 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 02:14:28.717154 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.716967 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 02:14:28.717154 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.717021 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-crd28\"" Nov 02 02:14:28.788193 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.788161 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7826eb46-3ec8-44d6-8340-219d0b34f09f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00b0cd5e-stnq4\" (UID: \"7826eb46-3ec8-44d6-8340-219d0b34f09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4" Nov 02 02:14:28.788391 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.788233 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7826eb46-3ec8-44d6-8340-219d0b34f09f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00b0cd5e-stnq4\" (UID: \"7826eb46-3ec8-44d6-8340-219d0b34f09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4" Nov 02 02:14:28.788483 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.788384 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf2rn\" (UniqueName: \"kubernetes.io/projected/7826eb46-3ec8-44d6-8340-219d0b34f09f-kube-api-access-wf2rn\") pod \"maas-api-revoke-sub-00b0cd5e-stnq4\" (UID: \"7826eb46-3ec8-44d6-8340-219d0b34f09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4" Nov 02 02:14:28.889526 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.889491 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7826eb46-3ec8-44d6-8340-219d0b34f09f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00b0cd5e-stnq4\" (UID: \"7826eb46-3ec8-44d6-8340-219d0b34f09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4" Nov 02 02:14:28.889761 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.889557 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf2rn\" (UniqueName: \"kubernetes.io/projected/7826eb46-3ec8-44d6-8340-219d0b34f09f-kube-api-access-wf2rn\") pod \"maas-api-revoke-sub-00b0cd5e-stnq4\" (UID: \"7826eb46-3ec8-44d6-8340-219d0b34f09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4" Nov 02 02:14:28.889761 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.889597 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7826eb46-3ec8-44d6-8340-219d0b34f09f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00b0cd5e-stnq4\" (UID: \"7826eb46-3ec8-44d6-8340-219d0b34f09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4" Nov 02 02:14:28.890306 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.890266 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7826eb46-3ec8-44d6-8340-219d0b34f09f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00b0cd5e-stnq4\" (UID: \"7826eb46-3ec8-44d6-8340-219d0b34f09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4" Nov 02 02:14:28.890566 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.890543 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:28.892073 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.892051 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7826eb46-3ec8-44d6-8340-219d0b34f09f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00b0cd5e-stnq4\" (UID: \"7826eb46-3ec8-44d6-8340-219d0b34f09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4" Nov 02 02:14:28.953321 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:28.953288 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf2rn\" (UniqueName: \"kubernetes.io/projected/7826eb46-3ec8-44d6-8340-219d0b34f09f-kube-api-access-wf2rn\") pod \"maas-api-revoke-sub-00b0cd5e-stnq4\" (UID: \"7826eb46-3ec8-44d6-8340-219d0b34f09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4" Nov 02 02:14:29.038706 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:29.038599 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:29.243295 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:29.243262 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4" Nov 02 02:14:29.616255 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:29.616225 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4"] Nov 02 02:14:29.617633 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:14:29.617611 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7826eb46_3ec8_44d6_8340_219d0b34f09f.slice/crio-efed8d4b7bdf31477c2e12ed3ebf15e1c371153b99c443db1038404f0f4cb7c2 WatchSource:0}: Error finding container efed8d4b7bdf31477c2e12ed3ebf15e1c371153b99c443db1038404f0f4cb7c2: Status 404 returned error can't find the container with id efed8d4b7bdf31477c2e12ed3ebf15e1c371153b99c443db1038404f0f4cb7c2 Nov 02 02:14:30.319368 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:30.319326 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4" event={"ID":"7826eb46-3ec8-44d6-8340-219d0b34f09f","Type":"ContainerStarted","Data":"d8072bf195bc1e51180144019f478cebfed2c0587fe230cf659ea92c30c08dd8"} Nov 02 02:14:30.319553 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:30.319392 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4" event={"ID":"7826eb46-3ec8-44d6-8340-219d0b34f09f","Type":"ContainerStarted","Data":"efed8d4b7bdf31477c2e12ed3ebf15e1c371153b99c443db1038404f0f4cb7c2"} Nov 02 02:14:30.438680 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:30.438622 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4" podStartSLOduration=2.438609585 podStartE2EDuration="2.438609585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:14:30.436854863 +0000 UTC m=+2987.108107544" watchObservedRunningTime="2026-11-02 02:14:30.438609585 +0000 UTC m=+2987.109862265" Nov 02 02:14:30.923066 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:30.923028 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:31.073863 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:31.073832 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:31.325385 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:31.325349 5044 generic.go:356] "Generic (PLEG): container finished" podID="7826eb46-3ec8-44d6-8340-219d0b34f09f" containerID="d8072bf195bc1e51180144019f478cebfed2c0587fe230cf659ea92c30c08dd8" exitCode=0 Nov 02 02:14:31.325572 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:31.325429 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4" event={"ID":"7826eb46-3ec8-44d6-8340-219d0b34f09f","Type":"ContainerDied","Data":"d8072bf195bc1e51180144019f478cebfed2c0587fe230cf659ea92c30c08dd8"} Nov 02 02:14:32.468235 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:32.468204 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4" Nov 02 02:14:32.625555 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:32.625484 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7826eb46-3ec8-44d6-8340-219d0b34f09f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7826eb46-3ec8-44d6-8340-219d0b34f09f-maas-api-xks-ca\") pod \"7826eb46-3ec8-44d6-8340-219d0b34f09f\" (UID: \"7826eb46-3ec8-44d6-8340-219d0b34f09f\") " Nov 02 02:14:32.625753 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:32.625557 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7826eb46-3ec8-44d6-8340-219d0b34f09f-kube-api-access-wf2rn\" (UniqueName: \"kubernetes.io/projected/7826eb46-3ec8-44d6-8340-219d0b34f09f-kube-api-access-wf2rn\") pod \"7826eb46-3ec8-44d6-8340-219d0b34f09f\" (UID: \"7826eb46-3ec8-44d6-8340-219d0b34f09f\") " Nov 02 02:14:32.625753 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:32.625612 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7826eb46-3ec8-44d6-8340-219d0b34f09f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7826eb46-3ec8-44d6-8340-219d0b34f09f-maas-api-openshift-ca\") pod \"7826eb46-3ec8-44d6-8340-219d0b34f09f\" (UID: \"7826eb46-3ec8-44d6-8340-219d0b34f09f\") " Nov 02 02:14:32.625990 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:32.625966 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7826eb46-3ec8-44d6-8340-219d0b34f09f-maas-api-openshift-ca" pod "7826eb46-3ec8-44d6-8340-219d0b34f09f" (UID: "7826eb46-3ec8-44d6-8340-219d0b34f09f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:14:32.627619 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:32.627597 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7826eb46-3ec8-44d6-8340-219d0b34f09f-maas-api-xks-ca" pod "7826eb46-3ec8-44d6-8340-219d0b34f09f" (UID: "7826eb46-3ec8-44d6-8340-219d0b34f09f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:14:32.627744 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:32.627699 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7826eb46-3ec8-44d6-8340-219d0b34f09f-kube-api-access-wf2rn" pod "7826eb46-3ec8-44d6-8340-219d0b34f09f" (UID: "7826eb46-3ec8-44d6-8340-219d0b34f09f"). InnerVolumeSpecName "kube-api-access-wf2rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:14:32.726855 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:32.726825 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7826eb46-3ec8-44d6-8340-219d0b34f09f-maas-api-xks-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:14:32.726855 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:32.726849 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf2rn\" (UniqueName: \"kubernetes.io/projected/7826eb46-3ec8-44d6-8340-219d0b34f09f-kube-api-access-wf2rn\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:14:32.726855 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:32.726860 5044 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7826eb46-3ec8-44d6-8340-219d0b34f09f-maas-api-openshift-ca\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:14:33.336644 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:33.336608 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4" event={"ID":"7826eb46-3ec8-44d6-8340-219d0b34f09f","Type":"ContainerDied","Data":"efed8d4b7bdf31477c2e12ed3ebf15e1c371153b99c443db1038404f0f4cb7c2"} Nov 02 02:14:33.336644 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:33.336644 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4" Nov 02 02:14:33.336894 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:33.336682 5044 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efed8d4b7bdf31477c2e12ed3ebf15e1c371153b99c443db1038404f0f4cb7c2" Nov 02 02:14:33.491549 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:33.491512 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr" podUID="152ed332-7e6a-4864-9763-726b003e585e" containerName="payload-processing" containerID="cri-o://f63264e0afe1e9c15d2203a9fc932414d64904c16919ad8f8d2e1de7e3af7c7f" gracePeriod=25 Nov 02 02:14:36.441272 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.441237 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" containerID="cri-o://7ac72594e91fb6d0bb61a26c231f86b11ae59b23e866a700ef3243ffb0809efa" gracePeriod=45 Nov 02 02:14:36.491230 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.491199 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" containerID="cri-o://6fb5591c86f2d36ec6cac463c67227b96f429e5ed6148dbc96f52a055b70b248" gracePeriod=45 Nov 02 02:14:36.508318 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.508289 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": dial tcp 10.132.0.62:8000: connect: connection refused" Nov 02 02:14:36.745479 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.745448 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 02:14:36.768477 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.768456 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 02:14:36.864635 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.864565 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-tls-certs\") pod \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " Nov 02 02:14:36.864635 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.864637 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-dshm\") pod \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " Nov 02 02:14:36.864898 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.864691 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-kube-api-access-ggfss\" (UniqueName: \"kubernetes.io/projected/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-kube-api-access-ggfss\") pod \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " Nov 02 02:14:36.864898 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.864728 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-model-cache\") pod \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " Nov 02 02:14:36.864898 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.864770 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-dshm\") pod \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " Nov 02 02:14:36.864898 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.864816 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-tmp-dir\") pod \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " Nov 02 02:14:36.864898 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.864876 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-model-cache\") pod \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " Nov 02 02:14:36.865177 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.864906 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-kube-api-access-vbv6g\" (UniqueName: \"kubernetes.io/projected/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-kube-api-access-vbv6g\") pod \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " Nov 02 02:14:36.865177 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.864948 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-tmp-dir\") pod \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " Nov 02 02:14:36.865177 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.864997 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-tls-certs\") pod \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " Nov 02 02:14:36.865177 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.865058 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-home\" (UniqueName: \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-home\") pod \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\" (UID: \"8c1ae51e-d552-4133-a0d5-5628c15cf7c7\") " Nov 02 02:14:36.865177 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.865105 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-home\" (UniqueName: \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-home\") pod \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\" (UID: \"3d7bd8f0-4f20-417d-8bdb-c6daecef3395\") " Nov 02 02:14:36.866106 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.866005 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-home" pod "3d7bd8f0-4f20-417d-8bdb-c6daecef3395" (UID: "3d7bd8f0-4f20-417d-8bdb-c6daecef3395"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:14:36.867649 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.867246 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-tmp-dir" pod "8c1ae51e-d552-4133-a0d5-5628c15cf7c7" (UID: "8c1ae51e-d552-4133-a0d5-5628c15cf7c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:14:36.867649 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.867272 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-model-cache" pod "3d7bd8f0-4f20-417d-8bdb-c6daecef3395" (UID: "3d7bd8f0-4f20-417d-8bdb-c6daecef3395"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:14:36.867649 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.867604 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-model-cache" pod "8c1ae51e-d552-4133-a0d5-5628c15cf7c7" (UID: "8c1ae51e-d552-4133-a0d5-5628c15cf7c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:14:36.867649 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.867586 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-home" pod "8c1ae51e-d552-4133-a0d5-5628c15cf7c7" (UID: "8c1ae51e-d552-4133-a0d5-5628c15cf7c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:14:36.868356 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.867861 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-tmp-dir" pod "3d7bd8f0-4f20-417d-8bdb-c6daecef3395" (UID: "3d7bd8f0-4f20-417d-8bdb-c6daecef3395"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:14:36.868356 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.868101 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-dshm" pod "3d7bd8f0-4f20-417d-8bdb-c6daecef3395" (UID: "3d7bd8f0-4f20-417d-8bdb-c6daecef3395"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:14:36.868762 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.868726 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-kube-api-access-vbv6g" pod "3d7bd8f0-4f20-417d-8bdb-c6daecef3395" (UID: "3d7bd8f0-4f20-417d-8bdb-c6daecef3395"). InnerVolumeSpecName "kube-api-access-vbv6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:14:36.869200 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.869173 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-dshm" pod "8c1ae51e-d552-4133-a0d5-5628c15cf7c7" (UID: "8c1ae51e-d552-4133-a0d5-5628c15cf7c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:14:36.870396 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.870367 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-kube-api-access-ggfss" pod "8c1ae51e-d552-4133-a0d5-5628c15cf7c7" (UID: "8c1ae51e-d552-4133-a0d5-5628c15cf7c7"). InnerVolumeSpecName "kube-api-access-ggfss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:14:36.871416 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.871385 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-tls-certs" pod "3d7bd8f0-4f20-417d-8bdb-c6daecef3395" (UID: "3d7bd8f0-4f20-417d-8bdb-c6daecef3395"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:14:36.872012 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.871989 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-tls-certs" pod "8c1ae51e-d552-4133-a0d5-5628c15cf7c7" (UID: "8c1ae51e-d552-4133-a0d5-5628c15cf7c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 02:14:36.966857 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.966779 5044 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-home\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:14:36.966857 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.966806 5044 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-home\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:14:36.966857 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.966814 5044 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-tls-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:14:36.966857 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.966823 5044 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-dshm\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:14:36.966857 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.966833 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggfss\" (UniqueName: \"kubernetes.io/projected/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-kube-api-access-ggfss\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:14:36.966857 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.966842 5044 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-model-cache\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:14:36.966857 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.966851 5044 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-dshm\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:14:36.967224 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.966878 5044 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-tmp-dir\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:14:36.967224 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.966887 5044 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-model-cache\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:14:36.967224 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.966895 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbv6g\" (UniqueName: \"kubernetes.io/projected/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-kube-api-access-vbv6g\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:14:36.967224 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.966904 5044 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d7bd8f0-4f20-417d-8bdb-c6daecef3395-tmp-dir\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:14:36.967224 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:36.966912 5044 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c1ae51e-d552-4133-a0d5-5628c15cf7c7-tls-certs\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:14:37.360915 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:37.360876 5044 generic.go:356] "Generic (PLEG): container finished" podID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerID="6fb5591c86f2d36ec6cac463c67227b96f429e5ed6148dbc96f52a055b70b248" exitCode=0 Nov 02 02:14:37.361069 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:37.360936 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" Nov 02 02:14:37.361069 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:37.360963 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" event={"ID":"8c1ae51e-d552-4133-a0d5-5628c15cf7c7","Type":"ContainerDied","Data":"6fb5591c86f2d36ec6cac463c67227b96f429e5ed6148dbc96f52a055b70b248"} Nov 02 02:14:37.361069 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:37.361010 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7" event={"ID":"8c1ae51e-d552-4133-a0d5-5628c15cf7c7","Type":"ContainerDied","Data":"de93d781a5f01f074259b569eaf0c92e98dba42e4eea13b8f18244377ad51113"} Nov 02 02:14:37.361069 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:37.361033 5044 scope.go:122] "RemoveContainer" containerID="6fb5591c86f2d36ec6cac463c67227b96f429e5ed6148dbc96f52a055b70b248" Nov 02 02:14:37.362473 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:37.362449 5044 generic.go:356] "Generic (PLEG): container finished" podID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerID="7ac72594e91fb6d0bb61a26c231f86b11ae59b23e866a700ef3243ffb0809efa" exitCode=0 Nov 02 02:14:37.362580 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:37.362489 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" event={"ID":"3d7bd8f0-4f20-417d-8bdb-c6daecef3395","Type":"ContainerDied","Data":"7ac72594e91fb6d0bb61a26c231f86b11ae59b23e866a700ef3243ffb0809efa"} Nov 02 02:14:37.362580 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:37.362511 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" event={"ID":"3d7bd8f0-4f20-417d-8bdb-c6daecef3395","Type":"ContainerDied","Data":"2a79c4c6bb1f5a07c2d10e878a3e2a030997d7cd1ee6e4870cb5530c3e15a020"} Nov 02 02:14:37.362580 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:37.362493 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv" Nov 02 02:14:37.370551 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:37.370531 5044 scope.go:122] "RemoveContainer" containerID="6fb5591c86f2d36ec6cac463c67227b96f429e5ed6148dbc96f52a055b70b248" Nov 02 02:14:37.370886 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:14:37.370864 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb5591c86f2d36ec6cac463c67227b96f429e5ed6148dbc96f52a055b70b248\": container with ID starting with 6fb5591c86f2d36ec6cac463c67227b96f429e5ed6148dbc96f52a055b70b248 not found: ID does not exist" containerID="6fb5591c86f2d36ec6cac463c67227b96f429e5ed6148dbc96f52a055b70b248" Nov 02 02:14:37.370963 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:37.370893 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb5591c86f2d36ec6cac463c67227b96f429e5ed6148dbc96f52a055b70b248"} err="failed to get container status \"6fb5591c86f2d36ec6cac463c67227b96f429e5ed6148dbc96f52a055b70b248\": rpc error: code = NotFound desc = could not find container \"6fb5591c86f2d36ec6cac463c67227b96f429e5ed6148dbc96f52a055b70b248\": container with ID starting with 6fb5591c86f2d36ec6cac463c67227b96f429e5ed6148dbc96f52a055b70b248 not found: ID does not exist" Nov 02 02:14:37.370963 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:37.370917 5044 scope.go:122] "RemoveContainer" containerID="7ac72594e91fb6d0bb61a26c231f86b11ae59b23e866a700ef3243ffb0809efa" Nov 02 02:14:37.379199 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:37.379181 5044 scope.go:122] "RemoveContainer" containerID="7ac72594e91fb6d0bb61a26c231f86b11ae59b23e866a700ef3243ffb0809efa" Nov 02 02:14:37.379434 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:14:37.379415 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ac72594e91fb6d0bb61a26c231f86b11ae59b23e866a700ef3243ffb0809efa\": container with ID starting with 7ac72594e91fb6d0bb61a26c231f86b11ae59b23e866a700ef3243ffb0809efa not found: ID does not exist" containerID="7ac72594e91fb6d0bb61a26c231f86b11ae59b23e866a700ef3243ffb0809efa" Nov 02 02:14:37.379470 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:37.379441 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac72594e91fb6d0bb61a26c231f86b11ae59b23e866a700ef3243ffb0809efa"} err="failed to get container status \"7ac72594e91fb6d0bb61a26c231f86b11ae59b23e866a700ef3243ffb0809efa\": rpc error: code = NotFound desc = could not find container \"7ac72594e91fb6d0bb61a26c231f86b11ae59b23e866a700ef3243ffb0809efa\": container with ID starting with 7ac72594e91fb6d0bb61a26c231f86b11ae59b23e866a700ef3243ffb0809efa not found: ID does not exist" Nov 02 02:14:37.528257 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:37.528220 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7"] Nov 02 02:14:37.579313 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:37.579288 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/simulator-w5-e06ea2-kserve-5b4c65f9fc-b5mb7"] Nov 02 02:14:37.691121 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:37.691050 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv"] Nov 02 02:14:37.696828 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:37.696805 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e06ea2/premium-w5-e06ea2-kserve-5bb5554468-xk4wv"] Nov 02 02:14:39.450741 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:39.450706 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" path="/var/lib/kubelet/pods/3d7bd8f0-4f20-417d-8bdb-c6daecef3395/volumes" Nov 02 02:14:39.451118 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:39.451101 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" path="/var/lib/kubelet/pods/8c1ae51e-d552-4133-a0d5-5628c15cf7c7/volumes" Nov 02 02:14:42.139141 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:42.139101 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr" podUID="152ed332-7e6a-4864-9763-726b003e585e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 02:14:43.183958 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:43.183925 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:43.839826 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:43.839797 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 02:14:43.849860 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:43.849841 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 02:14:44.734719 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:44.734685 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:45.015875 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:45.015845 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:45.639920 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:45.639882 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:50.289858 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:50.289821 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:51.874189 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:51.874157 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:52.020851 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:52.020802 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:52.138960 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:52.138886 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr" podUID="152ed332-7e6a-4864-9763-726b003e585e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 02:14:52.228362 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:52.228306 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:53.741473 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:53.741447 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr" Nov 02 02:14:53.816595 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:53.816564 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/152ed332-7e6a-4864-9763-726b003e585e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/152ed332-7e6a-4864-9763-726b003e585e-plugins-config-volume\") pod \"152ed332-7e6a-4864-9763-726b003e585e\" (UID: \"152ed332-7e6a-4864-9763-726b003e585e\") " Nov 02 02:14:53.816765 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:53.816653 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/152ed332-7e6a-4864-9763-726b003e585e-kube-api-access-rm4m5\" (UniqueName: \"kubernetes.io/projected/152ed332-7e6a-4864-9763-726b003e585e-kube-api-access-rm4m5\") pod \"152ed332-7e6a-4864-9763-726b003e585e\" (UID: \"152ed332-7e6a-4864-9763-726b003e585e\") " Nov 02 02:14:53.816935 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:53.816915 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/152ed332-7e6a-4864-9763-726b003e585e-plugins-config-volume" pod "152ed332-7e6a-4864-9763-726b003e585e" (UID: "152ed332-7e6a-4864-9763-726b003e585e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 02:14:53.818744 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:53.818723 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152ed332-7e6a-4864-9763-726b003e585e-kube-api-access-rm4m5" pod "152ed332-7e6a-4864-9763-726b003e585e" (UID: "152ed332-7e6a-4864-9763-726b003e585e"). InnerVolumeSpecName "kube-api-access-rm4m5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:14:53.917925 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:53.917897 5044 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/152ed332-7e6a-4864-9763-726b003e585e-plugins-config-volume\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:14:53.917925 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:53.917921 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm4m5\" (UniqueName: \"kubernetes.io/projected/152ed332-7e6a-4864-9763-726b003e585e-kube-api-access-rm4m5\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:14:54.171922 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:14:54.171833 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:14:54.452449 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:54.452366 5044 generic.go:356] "Generic (PLEG): container finished" podID="152ed332-7e6a-4864-9763-726b003e585e" containerID="f63264e0afe1e9c15d2203a9fc932414d64904c16919ad8f8d2e1de7e3af7c7f" exitCode=0 Nov 02 02:14:54.452449 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:54.452434 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr" Nov 02 02:14:54.452654 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:54.452450 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr" event={"ID":"152ed332-7e6a-4864-9763-726b003e585e","Type":"ContainerDied","Data":"f63264e0afe1e9c15d2203a9fc932414d64904c16919ad8f8d2e1de7e3af7c7f"} Nov 02 02:14:54.452654 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:54.452510 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr" event={"ID":"152ed332-7e6a-4864-9763-726b003e585e","Type":"ContainerDied","Data":"04c87fa084ee88e40fe09812fda8090341f60966723fc3047d240e75bab66486"} Nov 02 02:14:54.452654 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:54.452537 5044 scope.go:122] "RemoveContainer" containerID="f63264e0afe1e9c15d2203a9fc932414d64904c16919ad8f8d2e1de7e3af7c7f" Nov 02 02:14:54.461535 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:54.461519 5044 scope.go:122] "RemoveContainer" containerID="f63264e0afe1e9c15d2203a9fc932414d64904c16919ad8f8d2e1de7e3af7c7f" Nov 02 02:14:54.461859 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:14:54.461838 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63264e0afe1e9c15d2203a9fc932414d64904c16919ad8f8d2e1de7e3af7c7f\": container with ID starting with f63264e0afe1e9c15d2203a9fc932414d64904c16919ad8f8d2e1de7e3af7c7f not found: ID does not exist" containerID="f63264e0afe1e9c15d2203a9fc932414d64904c16919ad8f8d2e1de7e3af7c7f" Nov 02 02:14:54.461937 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:54.461873 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63264e0afe1e9c15d2203a9fc932414d64904c16919ad8f8d2e1de7e3af7c7f"} err="failed to get container status \"f63264e0afe1e9c15d2203a9fc932414d64904c16919ad8f8d2e1de7e3af7c7f\": rpc error: code = NotFound desc = could not find container \"f63264e0afe1e9c15d2203a9fc932414d64904c16919ad8f8d2e1de7e3af7c7f\": container with ID starting with f63264e0afe1e9c15d2203a9fc932414d64904c16919ad8f8d2e1de7e3af7c7f not found: ID does not exist" Nov 02 02:14:54.620774 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:54.620747 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr"] Nov 02 02:14:54.671880 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:54.671855 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e06ea2-7cc88ff567-5xvjr"] Nov 02 02:14:55.452772 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:55.452733 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="152ed332-7e6a-4864-9763-726b003e585e" path="/var/lib/kubelet/pods/152ed332-7e6a-4864-9763-726b003e585e/volumes" Nov 02 02:14:58.328474 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:58.328428 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:58.474830 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:58.474785 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:14:59.403952 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:14:59.403914 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:15:04.968460 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:04.968428 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:15:09.091872 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.091836 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h"] Nov 02 02:15:09.092246 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.092223 5044 cpu_manager.go:414] "containerMap: removing container" podUID="152ed332-7e6a-4864-9763-726b003e585e" containerName="payload-processing" Nov 02 02:15:09.092246 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.092234 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="152ed332-7e6a-4864-9763-726b003e585e" containerName="payload-processing" Nov 02 02:15:09.092317 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.092256 5044 cpu_manager.go:414] "containerMap: removing container" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" Nov 02 02:15:09.092317 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.092262 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" Nov 02 02:15:09.092317 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.092276 5044 cpu_manager.go:414] "containerMap: removing container" podUID="e16f31d3-d566-4620-a625-097e8ee9e28b" containerName="istio-proxy" Nov 02 02:15:09.092317 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.092282 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16f31d3-d566-4620-a625-097e8ee9e28b" containerName="istio-proxy" Nov 02 02:15:09.092317 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.092296 5044 cpu_manager.go:414] "containerMap: removing container" podUID="7826eb46-3ec8-44d6-8340-219d0b34f09f" containerName="revoke-keys" Nov 02 02:15:09.092317 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.092301 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7826eb46-3ec8-44d6-8340-219d0b34f09f" containerName="revoke-keys" Nov 02 02:15:09.092317 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.092307 5044 cpu_manager.go:414] "containerMap: removing container" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" Nov 02 02:15:09.092317 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.092311 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" Nov 02 02:15:09.092569 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.092369 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="152ed332-7e6a-4864-9763-726b003e585e" containerName="payload-processing" Nov 02 02:15:09.092569 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.092380 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7bd8f0-4f20-417d-8bdb-c6daecef3395" containerName="main" Nov 02 02:15:09.092569 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.092388 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="e16f31d3-d566-4620-a625-097e8ee9e28b" containerName="istio-proxy" Nov 02 02:15:09.092569 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.092395 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="652e50af-8a91-43ff-bb49-bbdf781feefd" containerName="istio-proxy" Nov 02 02:15:09.092569 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.092400 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1ae51e-d552-4133-a0d5-5628c15cf7c7" containerName="main" Nov 02 02:15:09.092569 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.092406 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="7826eb46-3ec8-44d6-8340-219d0b34f09f" containerName="revoke-keys" Nov 02 02:15:09.096883 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.096861 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.150476 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.150443 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pjf7g\"" Nov 02 02:15:09.156969 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.156942 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c38e6dce-aa89-4b1b-9f97-81d08429835d-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.157084 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.156978 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hfh4\" (UniqueName: \"kubernetes.io/projected/c38e6dce-aa89-4b1b-9f97-81d08429835d-kube-api-access-2hfh4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.157084 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.157006 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c38e6dce-aa89-4b1b-9f97-81d08429835d-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.157178 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.157091 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c38e6dce-aa89-4b1b-9f97-81d08429835d-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.157178 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.157119 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c38e6dce-aa89-4b1b-9f97-81d08429835d-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.157178 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.157165 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c38e6dce-aa89-4b1b-9f97-81d08429835d-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.157287 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.157260 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c38e6dce-aa89-4b1b-9f97-81d08429835d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.157324 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.157312 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c38e6dce-aa89-4b1b-9f97-81d08429835d-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.157366 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.157337 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c38e6dce-aa89-4b1b-9f97-81d08429835d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.208698 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.204718 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h"] Nov 02 02:15:09.258527 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.258499 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c38e6dce-aa89-4b1b-9f97-81d08429835d-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.258717 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.258536 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c38e6dce-aa89-4b1b-9f97-81d08429835d-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.258717 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.258558 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c38e6dce-aa89-4b1b-9f97-81d08429835d-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.258717 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.258593 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c38e6dce-aa89-4b1b-9f97-81d08429835d-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.258717 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.258618 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c38e6dce-aa89-4b1b-9f97-81d08429835d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.258717 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.258654 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c38e6dce-aa89-4b1b-9f97-81d08429835d-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.258717 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.258699 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c38e6dce-aa89-4b1b-9f97-81d08429835d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.259000 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.258740 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c38e6dce-aa89-4b1b-9f97-81d08429835d-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.259000 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.258779 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hfh4\" (UniqueName: \"kubernetes.io/projected/c38e6dce-aa89-4b1b-9f97-81d08429835d-kube-api-access-2hfh4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.259107 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.259011 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c38e6dce-aa89-4b1b-9f97-81d08429835d-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.259107 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.259068 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c38e6dce-aa89-4b1b-9f97-81d08429835d-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.259344 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.259312 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c38e6dce-aa89-4b1b-9f97-81d08429835d-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.259451 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.259404 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c38e6dce-aa89-4b1b-9f97-81d08429835d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.259451 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.259443 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c38e6dce-aa89-4b1b-9f97-81d08429835d-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.261041 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.261023 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c38e6dce-aa89-4b1b-9f97-81d08429835d-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.261297 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.261277 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c38e6dce-aa89-4b1b-9f97-81d08429835d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.321055 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.321026 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c38e6dce-aa89-4b1b-9f97-81d08429835d-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.321238 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.321218 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hfh4\" (UniqueName: \"kubernetes.io/projected/c38e6dce-aa89-4b1b-9f97-81d08429835d-kube-api-access-2hfh4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jlx9h\" (UID: \"c38e6dce-aa89-4b1b-9f97-81d08429835d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.412009 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.411935 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:09.838417 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.838391 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h"] Nov 02 02:15:09.840467 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:15:09.840431 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38e6dce_aa89_4b1b_9f97_81d08429835d.slice/crio-edf92dbf6fcfaa7225780ecca3738e54af28a1b594fa49ceb9633d7bec17879d WatchSource:0}: Error finding container edf92dbf6fcfaa7225780ecca3738e54af28a1b594fa49ceb9633d7bec17879d: Status 404 returned error can't find the container with id edf92dbf6fcfaa7225780ecca3738e54af28a1b594fa49ceb9633d7bec17879d Nov 02 02:15:09.842642 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.842607 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:15:09.842798 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.842703 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:15:09.842798 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:09.842749 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:15:10.533869 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:10.533821 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" event={"ID":"c38e6dce-aa89-4b1b-9f97-81d08429835d","Type":"ContainerStarted","Data":"90842edf975078c529968adf6fb3c4eb9e33cf4bc2ce20d7454163642bdea8cc"} Nov 02 02:15:10.533869 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:10.533875 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" event={"ID":"c38e6dce-aa89-4b1b-9f97-81d08429835d","Type":"ContainerStarted","Data":"edf92dbf6fcfaa7225780ecca3738e54af28a1b594fa49ceb9633d7bec17879d"} Nov 02 02:15:10.657974 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:10.657916 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" podStartSLOduration=1.657902768 podStartE2EDuration="1.657902768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:15:10.656269144 +0000 UTC m=+3027.327521839" watchObservedRunningTime="2026-11-02 02:15:10.657902768 +0000 UTC m=+3027.329155449" Nov 02 02:15:11.412159 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:11.412122 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:11.413921 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:11.413894 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 02 02:15:11.414063 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:11.413940 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" podUID="c38e6dce-aa89-4b1b-9f97-81d08429835d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 02 02:15:12.412465 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:12.412431 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 02 02:15:12.412877 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:12.412487 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" podUID="c38e6dce-aa89-4b1b-9f97-81d08429835d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 02 02:15:13.413333 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:13.413299 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 02 02:15:13.413736 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:13.413355 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" podUID="c38e6dce-aa89-4b1b-9f97-81d08429835d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 02 02:15:14.412934 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:14.412898 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 02 02:15:14.413114 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:14.412951 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" podUID="c38e6dce-aa89-4b1b-9f97-81d08429835d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 02 02:15:15.413332 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:15.413298 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 02 02:15:15.413782 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:15.413353 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" podUID="c38e6dce-aa89-4b1b-9f97-81d08429835d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 02 02:15:16.412959 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:16.412928 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 02 02:15:16.413145 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:16.412986 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" podUID="c38e6dce-aa89-4b1b-9f97-81d08429835d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 02 02:15:18.099997 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:18.099966 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:18.100970 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:18.100946 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:18.101729 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:18.101708 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" Nov 02 02:15:24.171972 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:15:24.171934 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:15:33.794522 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:33.794475 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:15:35.810894 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:35.810863 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:15:48.910064 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:48.910030 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:15:50.625595 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:15:50.625564 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:15:54.172460 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:15:54.172420 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:16:15.495918 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:15.495886 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:16:15.637471 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:15.637436 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:16:24.171935 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:16:24.171846 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:16:39.811336 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:39.811296 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd"] Nov 02 02:16:39.815211 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:39.815187 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:39.862345 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:39.862318 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl"] Nov 02 02:16:39.865719 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:39.865704 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:39.913374 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:39.913346 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:39.913497 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:39.913381 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:39.913497 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:39.913408 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:39.913497 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:39.913483 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpx8h\" (UniqueName: \"kubernetes.io/projected/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-kube-api-access-jpx8h\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:39.913687 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:39.913587 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:39.913687 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:39.913630 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:39.913776 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:39.913699 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:39.913832 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:39.913779 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:39.913893 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:39.913837 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:39.918536 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:39.918514 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd"] Nov 02 02:16:39.976096 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:39.976067 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl"] Nov 02 02:16:40.014683 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.014647 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:40.014847 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.014717 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:40.014847 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.014763 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96b2d140-e773-44cd-af96-e5600d34b15f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.014847 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.014795 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b2d140-e773-44cd-af96-e5600d34b15f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.014967 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.014898 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96b2d140-e773-44cd-af96-e5600d34b15f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.015031 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.015006 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pfs4\" (UniqueName: \"kubernetes.io/projected/96b2d140-e773-44cd-af96-e5600d34b15f-kube-api-access-7pfs4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.015071 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.015052 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:40.015071 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.015057 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:40.015149 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.015107 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:40.015235 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.015220 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:40.015301 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.015272 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:40.015357 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.015307 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:40.015357 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.015341 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96b2d140-e773-44cd-af96-e5600d34b15f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.015465 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.015383 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b2d140-e773-44cd-af96-e5600d34b15f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.015465 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.015450 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96b2d140-e773-44cd-af96-e5600d34b15f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.015565 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.015483 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:40.015603 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.015558 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpx8h\" (UniqueName: \"kubernetes.io/projected/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-kube-api-access-jpx8h\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:40.015648 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.015635 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:40.015744 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.015647 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:40.015744 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.015688 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96b2d140-e773-44cd-af96-e5600d34b15f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.015744 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.015714 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:40.015904 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.015742 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96b2d140-e773-44cd-af96-e5600d34b15f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.015942 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.015925 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:40.017505 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.017483 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:40.017940 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.017920 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:40.078066 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.078001 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:40.078066 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.078046 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpx8h\" (UniqueName: \"kubernetes.io/projected/a8c2294c-1f8b-42d4-8acf-8f2d2a2762db-kube-api-access-jpx8h\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-k28rd\" (UID: \"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:40.116744 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.116719 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96b2d140-e773-44cd-af96-e5600d34b15f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.116874 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.116755 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96b2d140-e773-44cd-af96-e5600d34b15f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.116874 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.116806 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96b2d140-e773-44cd-af96-e5600d34b15f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.116874 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.116839 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b2d140-e773-44cd-af96-e5600d34b15f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.116874 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.116867 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96b2d140-e773-44cd-af96-e5600d34b15f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.117095 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.116987 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pfs4\" (UniqueName: \"kubernetes.io/projected/96b2d140-e773-44cd-af96-e5600d34b15f-kube-api-access-7pfs4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.117156 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.117099 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96b2d140-e773-44cd-af96-e5600d34b15f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.117156 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.117150 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b2d140-e773-44cd-af96-e5600d34b15f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.117263 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.117185 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96b2d140-e773-44cd-af96-e5600d34b15f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.117263 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.117212 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96b2d140-e773-44cd-af96-e5600d34b15f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.117379 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.117276 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b2d140-e773-44cd-af96-e5600d34b15f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.117539 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.117514 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b2d140-e773-44cd-af96-e5600d34b15f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.117620 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.117520 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96b2d140-e773-44cd-af96-e5600d34b15f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.117796 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.117776 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96b2d140-e773-44cd-af96-e5600d34b15f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.118847 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.118824 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96b2d140-e773-44cd-af96-e5600d34b15f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.119170 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.119153 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96b2d140-e773-44cd-af96-e5600d34b15f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.128061 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.128044 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:40.180334 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.180307 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pfs4\" (UniqueName: \"kubernetes.io/projected/96b2d140-e773-44cd-af96-e5600d34b15f-kube-api-access-7pfs4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.180461 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.180309 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96b2d140-e773-44cd-af96-e5600d34b15f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2kgvl\" (UID: \"96b2d140-e773-44cd-af96-e5600d34b15f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.477842 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.477773 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:40.553561 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.553533 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd"] Nov 02 02:16:40.559169 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:16:40.559142 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c2294c_1f8b_42d4_8acf_8f2d2a2762db.slice/crio-e8526a495701cc09108b1895bb48c75643e8ebb470820f2a1c35e95311ade732 WatchSource:0}: Error finding container e8526a495701cc09108b1895bb48c75643e8ebb470820f2a1c35e95311ade732: Status 404 returned error can't find the container with id e8526a495701cc09108b1895bb48c75643e8ebb470820f2a1c35e95311ade732 Nov 02 02:16:40.561440 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.561405 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:16:40.561512 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.561474 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:16:40.561512 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.561503 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:16:40.855776 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.855742 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl"] Nov 02 02:16:40.860259 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:16:40.860231 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b2d140_e773_44cd_af96_e5600d34b15f.slice/crio-4bed1ad62a1974a8e0c1c3e4137fe6ca68b1ea22223208db4742d8ed58241700 WatchSource:0}: Error finding container 4bed1ad62a1974a8e0c1c3e4137fe6ca68b1ea22223208db4742d8ed58241700: Status 404 returned error can't find the container with id 4bed1ad62a1974a8e0c1c3e4137fe6ca68b1ea22223208db4742d8ed58241700 Nov 02 02:16:40.862664 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.862625 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:16:40.862810 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.862786 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:16:40.862860 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:40.862843 5044 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 02:16:41.002692 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:41.001449 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" event={"ID":"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db","Type":"ContainerStarted","Data":"f7ccc81663102168cfc1a95b4bf5b8f21ac287f8ede1ea7712639401541a64b7"} Nov 02 02:16:41.002692 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:41.001520 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" event={"ID":"a8c2294c-1f8b-42d4-8acf-8f2d2a2762db","Type":"ContainerStarted","Data":"e8526a495701cc09108b1895bb48c75643e8ebb470820f2a1c35e95311ade732"} Nov 02 02:16:41.003395 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:41.003356 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" event={"ID":"96b2d140-e773-44cd-af96-e5600d34b15f","Type":"ContainerStarted","Data":"269b2edd15ab46ce729f0dfe08bbd2f82c7b8c7ba88b9ee910821f05bfd9e923"} Nov 02 02:16:41.003491 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:41.003410 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" event={"ID":"96b2d140-e773-44cd-af96-e5600d34b15f","Type":"ContainerStarted","Data":"4bed1ad62a1974a8e0c1c3e4137fe6ca68b1ea22223208db4742d8ed58241700"} Nov 02 02:16:41.125623 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:41.125508 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" podStartSLOduration=2.125493056 podStartE2EDuration="2.125493056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:16:41.124186119 +0000 UTC m=+3117.795438800" watchObservedRunningTime="2026-11-02 02:16:41.125493056 +0000 UTC m=+3117.796745731" Nov 02 02:16:41.128878 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:41.128853 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:41.130340 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:41.130312 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-k28rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 02:16:41.130467 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:41.130352 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" podUID="a8c2294c-1f8b-42d4-8acf-8f2d2a2762db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 02:16:41.252026 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:41.251972 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" podStartSLOduration=2.251957441 podStartE2EDuration="2.251957441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:16:41.248925616 +0000 UTC m=+3117.920178295" watchObservedRunningTime="2026-11-02 02:16:41.251957441 +0000 UTC m=+3117.923210121" Nov 02 02:16:41.482751 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:41.481157 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:41.486691 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:41.483805 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 02 02:16:41.486691 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:41.483866 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" podUID="96b2d140-e773-44cd-af96-e5600d34b15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 02 02:16:42.129040 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:42.129008 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-k28rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 02:16:42.129526 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:42.129500 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" podUID="a8c2294c-1f8b-42d4-8acf-8f2d2a2762db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 02:16:42.484767 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:42.483759 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 02 02:16:42.484767 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:42.483816 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" podUID="96b2d140-e773-44cd-af96-e5600d34b15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 02 02:16:43.129293 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:43.129255 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-k28rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 02:16:43.129720 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:43.129341 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" podUID="a8c2294c-1f8b-42d4-8acf-8f2d2a2762db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 02:16:43.478768 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:43.478199 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 02 02:16:43.478768 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:43.478267 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" podUID="96b2d140-e773-44cd-af96-e5600d34b15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 02 02:16:43.695695 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:43.695201 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:16:44.129705 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:44.129331 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-k28rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 02:16:44.129705 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:44.129389 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" podUID="a8c2294c-1f8b-42d4-8acf-8f2d2a2762db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 02:16:44.490750 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:44.490459 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 02 02:16:44.490750 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:44.490523 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" podUID="96b2d140-e773-44cd-af96-e5600d34b15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 02 02:16:45.128562 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:45.128530 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-k28rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 02:16:45.128765 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:45.128584 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" podUID="a8c2294c-1f8b-42d4-8acf-8f2d2a2762db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 02:16:45.483859 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:45.483769 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 02 02:16:45.484437 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:45.484407 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" podUID="96b2d140-e773-44cd-af96-e5600d34b15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 02 02:16:46.128596 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:46.128554 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-k28rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 02:16:46.128832 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:46.128610 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" podUID="a8c2294c-1f8b-42d4-8acf-8f2d2a2762db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 02:16:46.485086 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:46.485005 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 02 02:16:46.485797 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:46.485765 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" podUID="96b2d140-e773-44cd-af96-e5600d34b15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 02 02:16:47.128709 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:47.128655 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-k28rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 02:16:47.128972 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:47.128725 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" podUID="a8c2294c-1f8b-42d4-8acf-8f2d2a2762db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 02:16:47.478898 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:47.478795 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 02 02:16:47.478898 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:47.478860 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" podUID="96b2d140-e773-44cd-af96-e5600d34b15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 02 02:16:48.128827 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:48.128788 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-k28rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 02 02:16:48.129334 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:48.128837 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" podUID="a8c2294c-1f8b-42d4-8acf-8f2d2a2762db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 02 02:16:49.479713 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:49.479662 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:16:49.480312 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:49.480276 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" podUID="96b2d140-e773-44cd-af96-e5600d34b15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:16:50.169884 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:50.169834 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-k28rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:16:50.170109 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:50.169935 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" podUID="a8c2294c-1f8b-42d4-8acf-8f2d2a2762db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:16:50.527837 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:50.527798 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:16:50.528294 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:50.528272 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" podUID="96b2d140-e773-44cd-af96-e5600d34b15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:16:50.990853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:50.990766 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:16:51.215841 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:51.215793 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-k28rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:16:51.216051 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:51.215888 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" podUID="a8c2294c-1f8b-42d4-8acf-8f2d2a2762db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:16:51.570828 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:51.570790 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:16:51.571332 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:51.570864 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" podUID="96b2d140-e773-44cd-af96-e5600d34b15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:16:52.216360 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:52.216319 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-k28rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:16:52.216661 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:52.216624 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" podUID="a8c2294c-1f8b-42d4-8acf-8f2d2a2762db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:16:52.571496 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:52.571464 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:16:52.572122 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:52.572092 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" podUID="96b2d140-e773-44cd-af96-e5600d34b15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:16:53.257889 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:53.257848 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-k28rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:16:53.258437 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:53.258130 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" podUID="a8c2294c-1f8b-42d4-8acf-8f2d2a2762db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:16:53.615206 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:53.615165 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:16:53.615943 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:53.615910 5044 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" podUID="96b2d140-e773-44cd-af96-e5600d34b15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:16:53.847714 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:53.847654 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:53.847989 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:53.847967 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:53.848713 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:53.848693 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" Nov 02 02:16:54.195008 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:16:54.194873 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:16:54.388808 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:54.388780 5044 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:54.389057 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:54.389041 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:16:54.389711 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:16:54.389695 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" Nov 02 02:17:24.172043 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:17:24.172007 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:17:51.336094 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:17:51.336062 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:17:54.171859 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:17:54.171775 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:18:24.172217 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:18:24.172174 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:18:53.060623 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:18:53.060591 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:18:54.171754 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:18:54.171720 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:19:24.172052 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:19:24.171952 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:19:32.057940 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:19:32.057908 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4"] Nov 02 02:19:32.109049 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:19:32.109013 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00b0cd5e-stnq4"] Nov 02 02:19:33.452700 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:19:33.452651 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7826eb46-3ec8-44d6-8340-219d0b34f09f" path="/var/lib/kubelet/pods/7826eb46-3ec8-44d6-8340-219d0b34f09f/volumes" Nov 02 02:19:41.357350 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:19:41.357314 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:19:43.873904 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:19:43.873868 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 02:19:43.882519 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:19:43.882497 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 02:19:52.079763 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:19:52.079720 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:19:54.172595 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:19:54.172558 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:20:24.174641 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:20:24.174592 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:20:34.136210 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:20:34.136161 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:20:34.136648 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:20:34.136247 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" podUID="c38e6dce-aa89-4b1b-9f97-81d08429835d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:20:44.752993 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:20:44.752963 5044 scope.go:122] "RemoveContainer" containerID="d8072bf195bc1e51180144019f478cebfed2c0587fe230cf659ea92c30c08dd8" Nov 02 02:20:52.452355 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:20:52.452322 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:20:54.171894 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:20:54.171854 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:21:09.284744 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:21:09.284706 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:21:24.174211 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:21:24.174175 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:21:52.763294 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:21:52.763259 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:21:54.172076 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:21:54.172039 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:22:24.171861 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:22:24.171785 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:22:34.914101 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:22:34.914062 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:22:35.501696 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:22:35.500150 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:22:35.706621 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:22:35.706584 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:22:40.061536 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:22:40.061499 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:22:40.237611 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:22:40.237578 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:22:40.395863 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:22:40.395777 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:22:46.333577 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:22:46.333542 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:22:46.488830 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:22:46.488791 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:22:46.636916 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:22:46.636831 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:22:51.273176 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:22:51.273121 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:22:51.871241 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:22:51.871208 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:22:52.043010 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:22:52.042982 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:22:53.096032 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:22:53.095998 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:22:54.176569 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:22:54.176527 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:22:56.217369 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:22:56.217337 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:23:00.990928 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:23:00.990892 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:23:04.944149 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:23:04.944107 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:23:05.550922 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:23:05.550895 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:23:05.703627 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:23:05.703593 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:23:05.846752 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:23:05.846638 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:23:06.114572 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:23:06.114491 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:23:10.003839 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:23:10.003803 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:23:10.989071 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:23:10.989042 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:23:15.785451 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:23:15.785414 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:23:20.493468 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:23:20.493426 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:23:24.172508 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:23:24.172468 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:23:25.270823 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:23:25.270782 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:23:30.067478 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:23:30.067445 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:23:34.976865 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:23:34.976830 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:23:39.790229 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:23:39.790183 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:23:44.596933 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:23:44.596867 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:23:49.378408 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:23:49.378370 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:23:53.437534 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:23:53.437445 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:23:54.171988 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:23:54.171952 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:23:54.176898 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:23:54.176875 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:23:58.883123 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:23:58.883043 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:24:03.676332 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:03.676290 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:24:05.016729 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:05.016666 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:24:08.722188 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:08.722155 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:24:10.072135 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:10.072103 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:24:13.664739 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:13.664708 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:24:17.403455 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:17.403393 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:24:17.543541 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:17.543511 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:24:18.378858 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:18.378828 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:24:22.557060 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:22.557026 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:24:24.059840 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:24.059806 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:24:24.176891 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:24:24.176850 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:24:24.335316 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:24.335225 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:24:25.081411 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:25.081377 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:24:30.822360 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:30.822322 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:24:30.973171 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:30.973141 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:24:31.186372 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:31.186291 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:24:43.904651 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:43.904618 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 02:24:43.915312 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:43.915292 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 02:24:46.942404 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:46.942356 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:24:47.213122 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:47.213047 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:24:52.774971 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:52.774939 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:24:53.781283 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:24:53.781253 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:24:54.173437 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:24:54.173347 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:25:07.316805 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:25:07.316770 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v"] Nov 02 02:25:07.317385 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:25:07.316989 5044 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v" podUID="1b919d07-7749-4e2c-ae2b-d18da91d92ef" containerName="manager" containerID="cri-o://dc6a1d6da10e869775ee6b94b26dda96768dbca2cc2dee87612be1ff98fbcbb5" gracePeriod=10 Nov 02 02:25:07.583796 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:25:07.583737 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v" Nov 02 02:25:07.658848 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:25:07.658816 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b919d07-7749-4e2c-ae2b-d18da91d92ef-kube-api-access-rmvsl\" (UniqueName: \"kubernetes.io/projected/1b919d07-7749-4e2c-ae2b-d18da91d92ef-kube-api-access-rmvsl\") pod \"1b919d07-7749-4e2c-ae2b-d18da91d92ef\" (UID: \"1b919d07-7749-4e2c-ae2b-d18da91d92ef\") " Nov 02 02:25:07.659027 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:25:07.658963 5044 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b919d07-7749-4e2c-ae2b-d18da91d92ef-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b919d07-7749-4e2c-ae2b-d18da91d92ef-extensions-socket-volume\") pod \"1b919d07-7749-4e2c-ae2b-d18da91d92ef\" (UID: \"1b919d07-7749-4e2c-ae2b-d18da91d92ef\") " Nov 02 02:25:07.659376 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:25:07.659353 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b919d07-7749-4e2c-ae2b-d18da91d92ef-extensions-socket-volume" pod "1b919d07-7749-4e2c-ae2b-d18da91d92ef" (UID: "1b919d07-7749-4e2c-ae2b-d18da91d92ef"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 02:25:07.661128 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:25:07.661100 5044 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b919d07-7749-4e2c-ae2b-d18da91d92ef-kube-api-access-rmvsl" pod "1b919d07-7749-4e2c-ae2b-d18da91d92ef" (UID: "1b919d07-7749-4e2c-ae2b-d18da91d92ef"). InnerVolumeSpecName "kube-api-access-rmvsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 02:25:07.760261 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:25:07.760225 5044 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b919d07-7749-4e2c-ae2b-d18da91d92ef-extensions-socket-volume\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:25:07.760261 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:25:07.760260 5044 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmvsl\" (UniqueName: \"kubernetes.io/projected/1b919d07-7749-4e2c-ae2b-d18da91d92ef-kube-api-access-rmvsl\") on node \"ip-10-0-129-6.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 02:25:07.870150 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:25:07.870062 5044 generic.go:356] "Generic (PLEG): container finished" podID="1b919d07-7749-4e2c-ae2b-d18da91d92ef" containerID="dc6a1d6da10e869775ee6b94b26dda96768dbca2cc2dee87612be1ff98fbcbb5" exitCode=0 Nov 02 02:25:07.870150 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:25:07.870134 5044 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v" Nov 02 02:25:07.870393 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:25:07.870154 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v" event={"ID":"1b919d07-7749-4e2c-ae2b-d18da91d92ef","Type":"ContainerDied","Data":"dc6a1d6da10e869775ee6b94b26dda96768dbca2cc2dee87612be1ff98fbcbb5"} Nov 02 02:25:07.870393 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:25:07.870221 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v" event={"ID":"1b919d07-7749-4e2c-ae2b-d18da91d92ef","Type":"ContainerDied","Data":"8e8e087cffd55df18f776039785e4fa185b437cd37504efcc916a25307bc5dde"} Nov 02 02:25:07.870393 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:25:07.870251 5044 scope.go:122] "RemoveContainer" containerID="dc6a1d6da10e869775ee6b94b26dda96768dbca2cc2dee87612be1ff98fbcbb5" Nov 02 02:25:07.882315 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:25:07.882299 5044 scope.go:122] "RemoveContainer" containerID="dc6a1d6da10e869775ee6b94b26dda96768dbca2cc2dee87612be1ff98fbcbb5" Nov 02 02:25:07.882566 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:25:07.882547 5044 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6a1d6da10e869775ee6b94b26dda96768dbca2cc2dee87612be1ff98fbcbb5\": container with ID starting with dc6a1d6da10e869775ee6b94b26dda96768dbca2cc2dee87612be1ff98fbcbb5 not found: ID does not exist" containerID="dc6a1d6da10e869775ee6b94b26dda96768dbca2cc2dee87612be1ff98fbcbb5" Nov 02 02:25:07.882616 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:25:07.882575 5044 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6a1d6da10e869775ee6b94b26dda96768dbca2cc2dee87612be1ff98fbcbb5"} err="failed to get container status \"dc6a1d6da10e869775ee6b94b26dda96768dbca2cc2dee87612be1ff98fbcbb5\": rpc error: code = NotFound desc = could not find container \"dc6a1d6da10e869775ee6b94b26dda96768dbca2cc2dee87612be1ff98fbcbb5\": container with ID starting with dc6a1d6da10e869775ee6b94b26dda96768dbca2cc2dee87612be1ff98fbcbb5 not found: ID does not exist" Nov 02 02:25:08.039042 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:25:08.039009 5044 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v"] Nov 02 02:25:08.042779 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:25:08.042750 5044 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-mw52v"] Nov 02 02:25:09.450725 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:25:09.450692 5044 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b919d07-7749-4e2c-ae2b-d18da91d92ef" path="/var/lib/kubelet/pods/1b919d07-7749-4e2c-ae2b-d18da91d92ef/volumes" Nov 02 02:25:24.172542 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:25:24.172464 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:25:54.159968 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:25:54.159917 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:25:54.172370 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:25:54.172340 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:26:10.948266 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:10.948234 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d7lf6"] Nov 02 02:26:10.948911 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:10.948886 5044 cpu_manager.go:414] "containerMap: removing container" podUID="1b919d07-7749-4e2c-ae2b-d18da91d92ef" containerName="manager" Nov 02 02:26:10.948911 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:10.948910 5044 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b919d07-7749-4e2c-ae2b-d18da91d92ef" containerName="manager" Nov 02 02:26:10.949045 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:10.949023 5044 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b919d07-7749-4e2c-ae2b-d18da91d92ef" containerName="manager" Nov 02 02:26:10.952211 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:10.952194 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d7lf6" Nov 02 02:26:11.056359 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:11.056325 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebe82eb4-675f-4668-8d9b-b86e51483a5b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-d7lf6\" (UID: \"ebe82eb4-675f-4668-8d9b-b86e51483a5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d7lf6" Nov 02 02:26:11.056521 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:11.056402 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2wn8\" (UniqueName: \"kubernetes.io/projected/ebe82eb4-675f-4668-8d9b-b86e51483a5b-kube-api-access-b2wn8\") pod \"kuadrant-operator-controller-manager-678d87766f-d7lf6\" (UID: \"ebe82eb4-675f-4668-8d9b-b86e51483a5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d7lf6" Nov 02 02:26:11.066406 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:11.066380 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xl9wm\"" Nov 02 02:26:11.068177 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:11.068158 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d7lf6"] Nov 02 02:26:11.157766 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:11.157726 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebe82eb4-675f-4668-8d9b-b86e51483a5b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-d7lf6\" (UID: \"ebe82eb4-675f-4668-8d9b-b86e51483a5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d7lf6" Nov 02 02:26:11.157766 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:11.157768 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2wn8\" (UniqueName: \"kubernetes.io/projected/ebe82eb4-675f-4668-8d9b-b86e51483a5b-kube-api-access-b2wn8\") pod \"kuadrant-operator-controller-manager-678d87766f-d7lf6\" (UID: \"ebe82eb4-675f-4668-8d9b-b86e51483a5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d7lf6" Nov 02 02:26:11.158069 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:11.158049 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebe82eb4-675f-4668-8d9b-b86e51483a5b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-d7lf6\" (UID: \"ebe82eb4-675f-4668-8d9b-b86e51483a5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d7lf6" Nov 02 02:26:11.220276 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:11.220213 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2wn8\" (UniqueName: \"kubernetes.io/projected/ebe82eb4-675f-4668-8d9b-b86e51483a5b-kube-api-access-b2wn8\") pod \"kuadrant-operator-controller-manager-678d87766f-d7lf6\" (UID: \"ebe82eb4-675f-4668-8d9b-b86e51483a5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d7lf6" Nov 02 02:26:11.264979 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:11.264946 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d7lf6" Nov 02 02:26:11.695132 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:11.695103 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d7lf6"] Nov 02 02:26:11.697515 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:26:11.697487 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe82eb4_675f_4668_8d9b_b86e51483a5b.slice/crio-393b240e857197d2ea524b1f38d1286367b47f21ca282959005d6c2c74fac655 WatchSource:0}: Error finding container 393b240e857197d2ea524b1f38d1286367b47f21ca282959005d6c2c74fac655: Status 404 returned error can't find the container with id 393b240e857197d2ea524b1f38d1286367b47f21ca282959005d6c2c74fac655 Nov 02 02:26:12.181234 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:12.181192 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d7lf6" event={"ID":"ebe82eb4-675f-4668-8d9b-b86e51483a5b","Type":"ContainerStarted","Data":"0b5b93fe7f9daa9f5b4e8a582468af9313ce8638f3bbeb67bbe36f5d66d7a4c3"} Nov 02 02:26:12.181627 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:12.181254 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d7lf6" event={"ID":"ebe82eb4-675f-4668-8d9b-b86e51483a5b","Type":"ContainerStarted","Data":"393b240e857197d2ea524b1f38d1286367b47f21ca282959005d6c2c74fac655"} Nov 02 02:26:12.181627 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:12.181374 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d7lf6" Nov 02 02:26:12.313340 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:12.313285 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d7lf6" podStartSLOduration=2.313270962 podStartE2EDuration="2.313270962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:26:12.312713536 +0000 UTC m=+3688.983966218" watchObservedRunningTime="2026-11-02 02:26:12.313270962 +0000 UTC m=+3688.984523642" Nov 02 02:26:23.189276 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:23.189245 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d7lf6" Nov 02 02:26:24.171704 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:26:24.171646 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:26:54.276755 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:26:54.266022 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:26:54.530832 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:54.527651 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:26:55.443809 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:55.443774 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:26:55.444215 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:26:55.443837 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" podUID="96b2d140-e773-44cd-af96-e5600d34b15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:27:16.754099 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:16.754064 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:17.491016 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:17.490857 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:17.675071 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:17.674784 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:21.807842 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:21.807802 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:21.973967 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:21.973933 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:22.126948 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:22.126833 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:24.172121 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:27:24.172013 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:27:28.109972 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:28.109922 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:28.271284 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:28.271248 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:28.421122 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:28.421030 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:33.080731 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:33.080698 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:33.650535 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:33.650504 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:33.824504 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:33.824477 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:37.996509 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:37.996472 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:42.874322 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:42.874284 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:46.657298 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:46.657252 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:47.285736 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:47.285706 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:47.538750 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:47.538663 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:47.685812 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:47.685779 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:48.229103 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:48.229064 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:51.097471 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:51.097440 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:53.014037 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:53.013992 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:54.172410 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:27:54.172376 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:27:54.846123 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:54.846089 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:27:57.771191 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:27:57.771159 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:28:02.572685 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:28:02.572627 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:28:07.394786 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:28:07.394750 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:28:12.178712 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:28:12.178649 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:28:16.979473 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:28:16.979439 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:28:21.692780 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:28:21.692745 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:28:24.172006 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:28:24.171926 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:28:26.480033 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:28:26.479994 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:28:31.181167 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:28:31.181130 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:28:35.982704 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:28:35.982660 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:28:36.385287 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:28:36.385255 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:28:41.083538 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:28:41.083499 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:28:41.446389 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:28:41.446299 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:28:45.888123 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:28:45.888085 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:28:48.725818 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:28:48.725789 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:28:48.869939 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:28:48.869907 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:28:50.668530 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:28:50.668494 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:28:54.172397 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:28:54.172364 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:28:54.861574 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:28:54.861531 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:28:55.187634 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:28:55.187542 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:28:56.364793 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:28:56.364758 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:28:56.646625 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:28:56.646538 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:28:57.422684 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:28:57.422645 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:29:03.436496 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:29:03.436464 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:29:03.579831 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:29:03.579796 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:29:03.782043 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:29:03.782009 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:29:10.135685 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:29:10.135649 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:29:18.223660 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:29:18.223621 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:29:18.715339 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:29:18.715307 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:29:24.170967 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:29:24.170921 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:29:33.424141 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:29:33.424107 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:29:42.342293 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:29:42.342248 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:29:42.492387 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:29:42.492352 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:29:43.936255 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:29:43.936225 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 02:29:43.947587 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:29:43.947565 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 02:29:52.026550 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:29:52.026511 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:29:54.172137 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:29:54.172053 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:29:54.187622 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:29:54.187596 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:29:54.332958 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:29:54.332928 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:29:55.539599 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:29:55.539561 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:30:24.174818 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:30:24.174778 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:30:35.369510 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:30:35.369449 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:30:39.398103 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:30:39.397999 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:30:39.554521 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:30:39.554491 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:30:39.891818 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:30:39.891768 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-k28rd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:30:39.892119 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:30:39.892087 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-k28rd" podUID="a8c2294c-1f8b-42d4-8acf-8f2d2a2762db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:30:40.437890 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:30:40.437824 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:30:40.438429 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:30:40.437932 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" podUID="96b2d140-e773-44cd-af96-e5600d34b15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:30:42.153839 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:30:42.153797 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:30:49.120062 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:30:49.120005 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:30:49.120611 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:30:49.120083 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" podUID="c38e6dce-aa89-4b1b-9f97-81d08429835d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:30:54.172246 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:30:54.172150 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:30:55.939225 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:30:55.939190 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:31:00.713695 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:00.711829 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:31:00.865901 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:00.865870 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:31:15.613738 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:15.613702 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:31:24.173713 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:31:24.173620 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:31:33.086316 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:33.086265 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:31:33.738610 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:33.738565 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:31:33.997045 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:33.996933 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:31:35.147033 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.146999 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw"] Nov 02 02:31:35.154125 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.154101 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw" Nov 02 02:31:35.209987 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.209954 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw"] Nov 02 02:31:35.259018 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.258987 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 02:31:35.259018 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.258987 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 02:31:35.259239 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.259022 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 02:31:35.259239 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.259100 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nk57n\"" Nov 02 02:31:35.259239 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.259161 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 02:31:35.320214 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.320182 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c9661f1-7f44-4cfa-9963-9532c1bf19b5-openshift-service-ca\") pod \"maas-api-644c59f4c-wpmfw\" (UID: \"3c9661f1-7f44-4cfa-9963-9532c1bf19b5\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw" Nov 02 02:31:35.320214 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.320216 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9xnq\" (UniqueName: \"kubernetes.io/projected/3c9661f1-7f44-4cfa-9963-9532c1bf19b5-kube-api-access-g9xnq\") pod \"maas-api-644c59f4c-wpmfw\" (UID: \"3c9661f1-7f44-4cfa-9963-9532c1bf19b5\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw" Nov 02 02:31:35.320414 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.320344 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9661f1-7f44-4cfa-9963-9532c1bf19b5-metrics-certs\") pod \"maas-api-644c59f4c-wpmfw\" (UID: \"3c9661f1-7f44-4cfa-9963-9532c1bf19b5\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw" Nov 02 02:31:35.320414 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.320373 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c9661f1-7f44-4cfa-9963-9532c1bf19b5-maas-api-tls\") pod \"maas-api-644c59f4c-wpmfw\" (UID: \"3c9661f1-7f44-4cfa-9963-9532c1bf19b5\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw" Nov 02 02:31:35.421789 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.421696 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c9661f1-7f44-4cfa-9963-9532c1bf19b5-openshift-service-ca\") pod \"maas-api-644c59f4c-wpmfw\" (UID: \"3c9661f1-7f44-4cfa-9963-9532c1bf19b5\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw" Nov 02 02:31:35.421789 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.421745 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9xnq\" (UniqueName: \"kubernetes.io/projected/3c9661f1-7f44-4cfa-9963-9532c1bf19b5-kube-api-access-g9xnq\") pod \"maas-api-644c59f4c-wpmfw\" (UID: \"3c9661f1-7f44-4cfa-9963-9532c1bf19b5\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw" Nov 02 02:31:35.421979 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.421826 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9661f1-7f44-4cfa-9963-9532c1bf19b5-metrics-certs\") pod \"maas-api-644c59f4c-wpmfw\" (UID: \"3c9661f1-7f44-4cfa-9963-9532c1bf19b5\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw" Nov 02 02:31:35.421979 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.421857 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c9661f1-7f44-4cfa-9963-9532c1bf19b5-maas-api-tls\") pod \"maas-api-644c59f4c-wpmfw\" (UID: \"3c9661f1-7f44-4cfa-9963-9532c1bf19b5\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw" Nov 02 02:31:35.422506 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.422477 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c9661f1-7f44-4cfa-9963-9532c1bf19b5-openshift-service-ca\") pod \"maas-api-644c59f4c-wpmfw\" (UID: \"3c9661f1-7f44-4cfa-9963-9532c1bf19b5\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw" Nov 02 02:31:35.424424 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.424394 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c9661f1-7f44-4cfa-9963-9532c1bf19b5-maas-api-tls\") pod \"maas-api-644c59f4c-wpmfw\" (UID: \"3c9661f1-7f44-4cfa-9963-9532c1bf19b5\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw" Nov 02 02:31:35.424527 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.424466 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9661f1-7f44-4cfa-9963-9532c1bf19b5-metrics-certs\") pod \"maas-api-644c59f4c-wpmfw\" (UID: \"3c9661f1-7f44-4cfa-9963-9532c1bf19b5\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw" Nov 02 02:31:35.484316 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.484292 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9xnq\" (UniqueName: \"kubernetes.io/projected/3c9661f1-7f44-4cfa-9963-9532c1bf19b5-kube-api-access-g9xnq\") pod \"maas-api-644c59f4c-wpmfw\" (UID: \"3c9661f1-7f44-4cfa-9963-9532c1bf19b5\") " pod="odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw" Nov 02 02:31:35.768444 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:35.768408 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw" Nov 02 02:31:36.145593 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:36.145566 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw"] Nov 02 02:31:36.147871 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:31:36.147842 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9661f1_7f44_4cfa_9963_9532c1bf19b5.slice/crio-fbea1c9c2869e4b8fe4de9913f574115f8dac4b4934b472db7eee95d6591ab44 WatchSource:0}: Error finding container fbea1c9c2869e4b8fe4de9913f574115f8dac4b4934b472db7eee95d6591ab44: Status 404 returned error can't find the container with id fbea1c9c2869e4b8fe4de9913f574115f8dac4b4934b472db7eee95d6591ab44 Nov 02 02:31:36.149378 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:36.149361 5044 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 02:31:36.831455 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:36.831420 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:31:36.995829 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:36.995805 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:31:37.000727 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:37.000664 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw" event={"ID":"3c9661f1-7f44-4cfa-9963-9532c1bf19b5","Type":"ContainerStarted","Data":"fbea1c9c2869e4b8fe4de9913f574115f8dac4b4934b472db7eee95d6591ab44"} Nov 02 02:31:37.143076 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:37.142985 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:31:38.007760 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:38.007713 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw" event={"ID":"3c9661f1-7f44-4cfa-9963-9532c1bf19b5","Type":"ContainerStarted","Data":"a248e8c661e7fe300c2e71ca9937b2bfd993a7c4af5a8c3b590c49276137d733"} Nov 02 02:31:38.008261 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:38.007876 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw" Nov 02 02:31:38.130960 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:38.130905 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw" podStartSLOduration=2.3211023060000002 podStartE2EDuration="3.130891532s" totalImagesPullingTime="809.789226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 02:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:31:38.129230378 +0000 UTC m=+4014.800483062" watchObservedRunningTime="2026-11-02 02:31:38.130891532 +0000 UTC m=+4014.802144213" Nov 02 02:31:41.513217 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:41.513179 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:31:41.677947 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:41.677913 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:31:41.820828 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:41.820735 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:31:44.019631 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:44.019602 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-644c59f4c-wpmfw" Nov 02 02:31:46.840032 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:46.840003 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:31:47.385470 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:47.385441 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:31:47.562282 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:47.562248 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:31:51.804124 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:51.804090 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:31:54.171806 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:31:54.171760 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:31:56.218533 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:56.218493 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:31:56.362849 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:56.362821 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:31:56.593913 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:31:56.593884 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:01.801664 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:01.801632 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:06.901287 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:06.901253 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:11.877652 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:11.877621 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:16.594771 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:16.594737 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:21.463308 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:21.463273 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:24.172996 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:32:24.172891 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:32:25.227896 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:25.227861 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:25.418208 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:25.418162 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:26.186430 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:26.186391 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:30.374028 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:30.373993 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:31.857500 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:31.857468 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:32.132500 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:32.132416 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:34.489647 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:34.489614 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:35.773411 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:35.773382 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:36.011855 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:36.011820 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:36.405633 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:36.405598 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:36.725691 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:36.725599 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:40.977450 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:40.977402 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:41.145089 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:41.145053 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:41.299173 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:41.299138 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:47.344180 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:47.344142 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:47.500095 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:47.500060 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:47.645919 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:47.645814 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:52.276760 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:52.276731 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:52.859438 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:52.859388 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:53.044891 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:53.044861 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:54.172076 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:32:54.172043 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:32:56.605871 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:56.605835 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:32:57.207321 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:32:57.207290 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:33:02.080946 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:33:02.080910 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:33:05.809945 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:33:05.809912 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:33:07.232416 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:33:07.232371 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:33:12.090376 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:33:12.090337 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:33:16.884191 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:33:16.884153 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:33:21.593902 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:33:21.593868 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:33:24.173485 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:33:24.173446 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:33:26.384087 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:33:26.384055 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:33:31.096206 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:33:31.096171 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:33:35.894513 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:33:35.894478 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:33:40.679281 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:33:40.679233 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:33:45.468944 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:33:45.468917 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:33:50.178112 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:33:50.178077 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:33:54.172399 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:33:54.172363 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:33:54.968852 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:33:54.968814 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:33:56.899792 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:33:56.899760 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:33:59.675790 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:33:59.675752 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:34:04.573892 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:34:04.573834 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:34:09.270708 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:34:09.270633 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:34:14.085424 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:34:14.085393 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:34:18.879880 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:34:18.879842 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:34:23.675850 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:34:23.675730 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:34:24.172560 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:34:24.172521 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:34:28.679763 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:34:28.679733 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:34:33.507893 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:34:33.507862 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:34:38.280385 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:34:38.280349 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:34:38.426527 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:34:38.426492 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:34:43.969853 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:34:43.969822 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 02:34:43.981394 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:34:43.981373 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-frfx9_7c285da3-2e11-498d-94f1-76c0ce604553/console-operator/2.log" Nov 02 02:34:47.528747 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:34:47.528712 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:34:49.012222 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:34:49.012185 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:34:49.296078 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:34:49.296002 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:34:50.078354 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:34:50.078324 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:34:54.172351 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:34:54.172302 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:34:55.946585 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:34:55.946542 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:34:56.093837 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:34:56.093789 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:34:56.302287 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:34:56.302255 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:34:57.252724 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:34:57.252686 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:35:04.146845 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:35:04.146804 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:35:04.147363 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:35:04.146882 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jlx9h" podUID="c38e6dce-aa89-4b1b-9f97-81d08429835d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:35:24.171646 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:35:24.171610 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:35:40.433534 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:35:40.432848 5044 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 02:35:40.433534 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:35:40.432976 5044 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2kgvl" podUID="96b2d140-e773-44cd-af96-e5600d34b15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 02:35:54.171535 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:35:54.171451 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:35:57.666711 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:35:57.666640 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:36:16.027120 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:16.027087 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-684875949c-whjhb_2a9d8c11-36a4-4fce-828e-86089d79dba4/manager/0.log" Nov 02 02:36:17.768752 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:17.768723 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-xvcfj_f14b189b-bc74-4186-b2b5-c20db167e680/server/0.log" Nov 02 02:36:17.996967 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:17.996938 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-s7zvr_9a000a5f-a242-4f5c-a7ee-335e84671c33/manager/0.log" Nov 02 02:36:19.463396 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:19.463368 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-644c59f4c-wpmfw_3c9661f1-7f44-4cfa-9963-9532c1bf19b5/maas-api/0.log" Nov 02 02:36:22.460560 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:22.460525 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-dzndg_686cd2a7-b05e-45c4-af71-d0d7a6997a8c/manager/0.log" Nov 02 02:36:23.249238 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:23.249208 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-d7lf6_ebe82eb4-675f-4668-8d9b-b86e51483a5b/manager/0.log" Nov 02 02:36:24.078708 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:24.078662 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-7c9h7_bafa8ecd-2b0c-4f80-b065-edabb802652f/manager/0.log" Nov 02 02:36:24.171785 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:36:24.171756 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:36:25.145660 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:25.145615 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-n7g9x_e9655a38-bc2f-498e-b3ea-b2d17e408844/discovery/0.log" Nov 02 02:36:25.929625 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:25.929570 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-2kgvl_96b2d140-e773-44cd-af96-e5600d34b15f/istio-proxy/0.log" Nov 02 02:36:26.656165 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:26.656134 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-jlx9h_c38e6dce-aa89-4b1b-9f97-81d08429835d/istio-proxy/0.log" Nov 02 02:36:26.931391 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:26.931313 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-k28rd_a8c2294c-1f8b-42d4-8acf-8f2d2a2762db/istio-proxy/0.log" Nov 02 02:36:28.384897 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:28.384869 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2_a9d93929-7a30-4a2a-a145-9869ea9b44fe/istio-proxy/0.log" Nov 02 02:36:29.119833 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:29.119806 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cb67dcd4c-zk2vs_bdebb5b6-76a2-48c8-810d-6f80549a4f3c/router/0.log" Nov 02 02:36:30.764414 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:30.764384 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-fg2fj_67c587ab-de84-4d10-a0de-69762a529054/main/0.log" Nov 02 02:36:31.287888 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:31.287852 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-p9dmt_0981d7f6-9826-4e4d-a643-09a2514967d0/main/0.log" Nov 02 02:36:31.845077 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:31.845004 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4_39c708b5-209a-4b5c-9f5b-402174f822b4/main/0.log" Nov 02 02:36:42.146357 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:42.146325 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-684875949c-whjhb_2a9d8c11-36a4-4fce-828e-86089d79dba4/manager/0.log" Nov 02 02:36:43.724029 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:43.724000 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-xvcfj_f14b189b-bc74-4186-b2b5-c20db167e680/server/0.log" Nov 02 02:36:43.951606 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:43.951573 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-s7zvr_9a000a5f-a242-4f5c-a7ee-335e84671c33/manager/0.log" Nov 02 02:36:45.272963 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:45.272927 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-644c59f4c-wpmfw_3c9661f1-7f44-4cfa-9963-9532c1bf19b5/maas-api/0.log" Nov 02 02:36:47.940478 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:47.940445 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-dzndg_686cd2a7-b05e-45c4-af71-d0d7a6997a8c/manager/0.log" Nov 02 02:36:48.729904 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:48.729873 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-d7lf6_ebe82eb4-675f-4668-8d9b-b86e51483a5b/manager/0.log" Nov 02 02:36:49.389265 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:49.389229 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-7c9h7_bafa8ecd-2b0c-4f80-b065-edabb802652f/manager/0.log" Nov 02 02:36:50.298983 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:50.298952 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-n7g9x_e9655a38-bc2f-498e-b3ea-b2d17e408844/discovery/0.log" Nov 02 02:36:50.997335 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:50.997305 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-2kgvl_96b2d140-e773-44cd-af96-e5600d34b15f/istio-proxy/0.log" Nov 02 02:36:51.739286 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:51.739260 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-jlx9h_c38e6dce-aa89-4b1b-9f97-81d08429835d/istio-proxy/0.log" Nov 02 02:36:52.011857 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:52.011830 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-k28rd_a8c2294c-1f8b-42d4-8acf-8f2d2a2762db/istio-proxy/0.log" Nov 02 02:36:53.428523 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:53.428484 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64487b45b7-vtzj2_a9d93929-7a30-4a2a-a145-9869ea9b44fe/istio-proxy/0.log" Nov 02 02:36:54.153520 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:54.153494 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6cb67dcd4c-zk2vs_bdebb5b6-76a2-48c8-810d-6f80549a4f3c/router/0.log" Nov 02 02:36:54.172573 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:36:54.172541 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:36:55.573781 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:55.573751 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-fg2fj_67c587ab-de84-4d10-a0de-69762a529054/main/0.log" Nov 02 02:36:56.047096 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:56.047066 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-p9dmt_0981d7f6-9826-4e4d-a643-09a2514967d0/main/0.log" Nov 02 02:36:56.543889 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:56.543858 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-rshf4_39c708b5-209a-4b5c-9f5b-402174f822b4/main/0.log" Nov 02 02:36:57.920917 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:36:57.920888 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:37:24.174099 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:37:24.173976 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:37:46.352785 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:37:46.352741 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:37:54.172305 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:37:54.172263 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:37:58.283368 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:37:58.283335 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:38:04.191621 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:04.191591 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:38:04.341930 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:04.341894 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:38:13.609273 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:13.609238 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7szx6_79f97db3-814b-4849-ab8d-305dbcd3449c/global-pull-secret-syncer/0.log" Nov 02 02:38:15.461624 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:15.461598 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-v8sdv_eb8ad2e6-7e4f-4c0a-9ea3-99ed98344ba6/connection-checker/0.log" Nov 02 02:38:15.890603 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:15.890580 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xmlts_8618e1e8-4d71-4f7a-b2a1-1b950190f110/connection-checker/0.log" Nov 02 02:38:16.343638 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:16.343611 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-zlq59_2e031e5c-cb8a-42b9-b334-1cbe61a9c59f/connection-checker/0.log" Nov 02 02:38:16.791271 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:16.791239 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" Nov 02 02:38:18.136363 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:18.136327 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-6.us-west-1.compute.internal_817dd2f8fadd21a6e3346c963f0534ec/haproxy/0.log" Nov 02 02:38:18.369064 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:18.369032 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:38:18.517109 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:18.517073 5044 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7mk45_bb2a916a-fc15-4bdc-b28a-c2473a543ce5/konnectivity-agent/0.log" limit=1024 Nov 02 02:38:24.067226 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.067193 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-dzndg_686cd2a7-b05e-45c4-af71-d0d7a6997a8c/manager/0.log" Nov 02 02:38:24.172379 ip-10-0-129-6 kubenswrapper[5044]: E1102 02:38:24.172341 5044 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2hxq" containerName="extractor" Nov 02 02:38:24.440369 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.440284 5044 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq"] Nov 02 02:38:24.444022 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.444002 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" Nov 02 02:38:24.549861 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.549832 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zt4ds\"/\"openshift-service-ca.crt\"" Nov 02 02:38:24.550060 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.549865 5044 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zt4ds\"/\"default-dockercfg-kqrg4\"" Nov 02 02:38:24.550060 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.549996 5044 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zt4ds\"/\"kube-root-ca.crt\"" Nov 02 02:38:24.552106 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.552080 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq"] Nov 02 02:38:24.575359 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.575338 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e3da745-200c-4217-9f20-619f9b9d4fc7-sys\") pod \"perf-node-gather-daemonset-j4tgq\" (UID: \"1e3da745-200c-4217-9f20-619f9b9d4fc7\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" Nov 02 02:38:24.575464 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.575368 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e3da745-200c-4217-9f20-619f9b9d4fc7-lib-modules\") pod \"perf-node-gather-daemonset-j4tgq\" (UID: \"1e3da745-200c-4217-9f20-619f9b9d4fc7\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" Nov 02 02:38:24.575464 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.575411 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e3da745-200c-4217-9f20-619f9b9d4fc7-proc\") pod \"perf-node-gather-daemonset-j4tgq\" (UID: \"1e3da745-200c-4217-9f20-619f9b9d4fc7\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" Nov 02 02:38:24.575464 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.575460 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbg8v\" (UniqueName: \"kubernetes.io/projected/1e3da745-200c-4217-9f20-619f9b9d4fc7-kube-api-access-fbg8v\") pod \"perf-node-gather-daemonset-j4tgq\" (UID: \"1e3da745-200c-4217-9f20-619f9b9d4fc7\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" Nov 02 02:38:24.575599 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.575585 5044 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e3da745-200c-4217-9f20-619f9b9d4fc7-podres\") pod \"perf-node-gather-daemonset-j4tgq\" (UID: \"1e3da745-200c-4217-9f20-619f9b9d4fc7\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" Nov 02 02:38:24.676251 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.676225 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e3da745-200c-4217-9f20-619f9b9d4fc7-sys\") pod \"perf-node-gather-daemonset-j4tgq\" (UID: \"1e3da745-200c-4217-9f20-619f9b9d4fc7\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" Nov 02 02:38:24.676361 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.676257 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e3da745-200c-4217-9f20-619f9b9d4fc7-lib-modules\") pod \"perf-node-gather-daemonset-j4tgq\" (UID: \"1e3da745-200c-4217-9f20-619f9b9d4fc7\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" Nov 02 02:38:24.676361 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.676316 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e3da745-200c-4217-9f20-619f9b9d4fc7-proc\") pod \"perf-node-gather-daemonset-j4tgq\" (UID: \"1e3da745-200c-4217-9f20-619f9b9d4fc7\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" Nov 02 02:38:24.676361 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.676349 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e3da745-200c-4217-9f20-619f9b9d4fc7-sys\") pod \"perf-node-gather-daemonset-j4tgq\" (UID: \"1e3da745-200c-4217-9f20-619f9b9d4fc7\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" Nov 02 02:38:24.676361 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.676351 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbg8v\" (UniqueName: \"kubernetes.io/projected/1e3da745-200c-4217-9f20-619f9b9d4fc7-kube-api-access-fbg8v\") pod \"perf-node-gather-daemonset-j4tgq\" (UID: \"1e3da745-200c-4217-9f20-619f9b9d4fc7\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" Nov 02 02:38:24.676500 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.676435 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e3da745-200c-4217-9f20-619f9b9d4fc7-lib-modules\") pod \"perf-node-gather-daemonset-j4tgq\" (UID: \"1e3da745-200c-4217-9f20-619f9b9d4fc7\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" Nov 02 02:38:24.676500 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.676452 5044 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e3da745-200c-4217-9f20-619f9b9d4fc7-podres\") pod \"perf-node-gather-daemonset-j4tgq\" (UID: \"1e3da745-200c-4217-9f20-619f9b9d4fc7\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" Nov 02 02:38:24.676565 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.676530 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e3da745-200c-4217-9f20-619f9b9d4fc7-podres\") pod \"perf-node-gather-daemonset-j4tgq\" (UID: \"1e3da745-200c-4217-9f20-619f9b9d4fc7\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" Nov 02 02:38:24.676621 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.676607 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e3da745-200c-4217-9f20-619f9b9d4fc7-proc\") pod \"perf-node-gather-daemonset-j4tgq\" (UID: \"1e3da745-200c-4217-9f20-619f9b9d4fc7\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" Nov 02 02:38:24.738274 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.738213 5044 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbg8v\" (UniqueName: \"kubernetes.io/projected/1e3da745-200c-4217-9f20-619f9b9d4fc7-kube-api-access-fbg8v\") pod \"perf-node-gather-daemonset-j4tgq\" (UID: \"1e3da745-200c-4217-9f20-619f9b9d4fc7\") " pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" Nov 02 02:38:24.756985 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:24.756964 5044 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" Nov 02 02:38:25.127875 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:25.127851 5044 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq"] Nov 02 02:38:25.129498 ip-10-0-129-6 kubenswrapper[5044]: W1102 02:38:25.129463 5044 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1e3da745_200c_4217_9f20_619f9b9d4fc7.slice/crio-17fdb2bbd5e42817257dd38e1f519d172b6072ce9b1c3d9b7178598662f76690 WatchSource:0}: Error finding container 17fdb2bbd5e42817257dd38e1f519d172b6072ce9b1c3d9b7178598662f76690: Status 404 returned error can't find the container with id 17fdb2bbd5e42817257dd38e1f519d172b6072ce9b1c3d9b7178598662f76690 Nov 02 02:38:25.388498 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:25.388409 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-d7lf6_ebe82eb4-675f-4668-8d9b-b86e51483a5b/manager/0.log" Nov 02 02:38:26.108034 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:26.108001 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" event={"ID":"1e3da745-200c-4217-9f20-619f9b9d4fc7","Type":"ContainerStarted","Data":"c13130eb4db3d3b920ff8b79aeb60e05f0ed2d49b684c5d6dfddeadab82e814f"} Nov 02 02:38:26.108211 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:26.108055 5044 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" event={"ID":"1e3da745-200c-4217-9f20-619f9b9d4fc7","Type":"ContainerStarted","Data":"17fdb2bbd5e42817257dd38e1f519d172b6072ce9b1c3d9b7178598662f76690"} Nov 02 02:38:26.108211 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:26.108124 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" Nov 02 02:38:26.223275 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:26.223246 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-7c9h7_bafa8ecd-2b0c-4f80-b065-edabb802652f/manager/0.log" Nov 02 02:38:26.226350 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:26.226307 5044 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" podStartSLOduration=2.22629506 podStartE2EDuration="2.22629506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 02:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 02:38:26.225833222 +0000 UTC m=+4422.897085904" watchObservedRunningTime="2026-11-02 02:38:26.22629506 +0000 UTC m=+4422.897547740" Nov 02 02:38:31.705916 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:31.705886 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-6sdmz_59ccadf3-202d-481d-8c8b-c4fa86c92b8b/cluster-monitoring-operator/0.log" Nov 02 02:38:32.123655 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:32.123625 5044 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zt4ds/perf-node-gather-daemonset-j4tgq" Nov 02 02:38:36.809421 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:36.809386 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z9mp6_928ee3de-8fae-4a90-8fea-84b30d084700/node-exporter/0.log" Nov 02 02:38:37.225906 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:37.225835 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z9mp6_928ee3de-8fae-4a90-8fea-84b30d084700/kube-rbac-proxy/0.log" Nov 02 02:38:37.647388 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:37.647358 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z9mp6_928ee3de-8fae-4a90-8fea-84b30d084700/init-textfile/0.log" Nov 02 02:38:43.082403 ip-10-0-129-6 kubenswrapper[5044]: I1102 02:38:43.082376 5044 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-kd7nc_d75a9f61-49b8-4bf0-b3e5-4200e3ec7332/prometheus-operator-admission-webhook/0.log"